Skip to content

hooks@0.3.29

Latest

Choose a tag to compare

@gram-bot gram-bot released this 31 Aug 21:02
· 66 commits to main since this release
90f2703
feat: generate skill recommendations passively (#5838)

## Summary

Replace agent-invoked skill feedback tools with passive recommendations
generated by the existing efficacy analysis. High-confidence,
transcript-grounded recommendations enter the current feedback and
consolidation workflow, while a recommendation-aware benchmark measures
candidate judge models without exposing private session content.

## Motivation

The bundled feedback MCP depended on agents actively reporting skill
failures and was invoked infrequently. Efficacy analysis already
observes the relevant session evidence and can produce structured
feedback without adding another agent-facing tool.

## Impact

New plugin packages and managed assistants no longer expose the
skill-feedback tool. Previously installed clients can continue
submitting feedback through the retained hooks endpoint.

Passive feedback remains restricted to high-confidence recommendations,
but the benchmark shows that none of the evaluated model configurations
currently satisfies every score, positive-emission, suppression, pair,
and persistence gate. The production model is unchanged, and the
benchmark provides an explicit guard for subsequent model or prompt
selection.

## Technical details

### Passive recommendation generation

The efficacy judge returns non-positive recommendations with a closed
issue classification, a closed change classification, and direct
transcript evidence indices. Only structurally valid, high-confidence
recommendations are persisted and attributed to the evaluated skill
version and session.

### Persistence safety

Recommendation notes are redacted for recognized secrets and normalized
before PostgreSQL persistence. Feedback identifiers derive from stable
recommendation content, retries remain idempotent, and persisted
recommendations continue through the existing suggestion workflow.

### Recommendation-aware benchmark

The synthetic corpus contains ten score cases and five compact
pre-correction/post-correction pairs. Pre-correction probes expect the
hidden corrective recommendation; post-correction probes require silence
for the same problem. Deterministic grading checks count, outcome,
confidence, issue, change, and transcript grounding without using
another LLM or serializing notes, labels, evidence indices, transcripts,
or skill bodies.

The three-run v11 comparison found Terra strongest on recommendation and
pair agreement, Gemini lowest-latency, and GLM cheapest with the
strongest suppression. No candidate passed every rollout gate.

### Compatibility

The hooks RPC handler and SDK contract remain available for lagging
clients, while new generated plugin configurations omit the stdio
sidecar and its dedicated hooks credential.