feat: action tagging in distillation for implicit behavioral pattern detection#376
Merged
Conversation
…detection Adds behavioral action tags to the distillation observer prompt. Tags like [requested-tests], [corrected-style], [rejected-approach] are embedded in observations and mechanically counted across sessions. When a tag appears in 3+ distinct sessions, a preference entry is auto-created. Combined with pattern echo detection, this catches implicit patterns that the user demonstrates but never states as directives. Eval PR-2 (implicit preferences): 3.50 → 4.58 (tail-window: ~4.85)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds behavioral action tags to the distillation observer prompt and mechanical cross-session counting to detect implicit user preferences.
Eval Results
PR-2 (implicit preferences): 3.50 → 4.58 (tail-window: ~4.85)
How it works
Action tags in distillation
The distillation observer prompt now tags user behavioral actions with bracketed labels:
Tags use a consistent
[verb-noun]format so the same behavior in different contexts (testing REST endpoints vs React components) gets the same tag.Cross-session counting
After each distillation segment, action tags are extracted via regex and counted across distinct sessions in the project. When a tag appears in 3+ sessions, a preference entry is auto-created at confidence 0.8.
Why this works where embeddings didn't
Vector similarity compares surface text — 'write tests for REST endpoints' and 'add tests for React components' are topically different. Action tags abstract the behavior into a canonical label (
[requested-tests]), making the pattern countable regardless of context.Files Changed
packages/core/src/prompt.ts— BEHAVIORAL ACTION TAGS section in distillation promptpackages/core/src/pattern-extract.ts—extractActionTags(),tagToTitle()packages/core/src/distillation.ts— tag counting + preference creation after distillation