You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
feat: generate skill recommendations passively (#5838)
## Summary
Replace agent-invoked skill feedback tools with passive recommendations
generated by the existing efficacy analysis. High-confidence,
transcript-grounded recommendations enter the current feedback and
consolidation workflow, while a recommendation-aware benchmark measures
candidate judge models without exposing private session content.
## Motivation
The bundled feedback MCP depended on agents actively reporting skill
failures and was invoked infrequently. Efficacy analysis already
observes the relevant session evidence and can produce structured
feedback without adding another agent-facing tool.
## Impact
New plugin packages and managed assistants no longer expose the
skill-feedback tool. Previously installed clients can continue
submitting feedback through the retained hooks endpoint.
Passive feedback remains restricted to high-confidence recommendations,
but the benchmark shows that none of the evaluated model configurations
currently satisfies every score, positive-emission, suppression, pair,
and persistence gate. The production model is unchanged, and the
benchmark provides an explicit guard for subsequent model or prompt
selection.
## Technical details
### Passive recommendation generation
The efficacy judge returns non-positive recommendations with a closed
issue classification, a closed change classification, and direct
transcript evidence indices. Only structurally valid, high-confidence
recommendations are persisted and attributed to the evaluated skill
version and session.
### Persistence safety
Recommendation notes are redacted for recognized secrets and normalized
before PostgreSQL persistence. Feedback identifiers derive from stable
recommendation content, retries remain idempotent, and persisted
recommendations continue through the existing suggestion workflow.
### Recommendation-aware benchmark
The synthetic corpus contains ten score cases and five compact
pre-correction/post-correction pairs. Pre-correction probes expect the
hidden corrective recommendation; post-correction probes require silence
for the same problem. Deterministic grading checks count, outcome,
confidence, issue, change, and transcript grounding without using
another LLM or serializing notes, labels, evidence indices, transcripts,
or skill bodies.
The three-run v11 comparison found Terra strongest on recommendation and
pair agreement, Gemini lowest-latency, and GLM cheapest with the
strongest suppression. No candidate passed every rollout gate.
### Compatibility
The hooks RPC handler and SDK contract remain available for lagging
clients, while new generated plugin configurations omit the stdio
sidecar and its dedicated hooks credential.