perf(tokens): autoresearch loop 99.2% reduction (5513→42)

[Gradata]

Summary

• 100-iteration autoresearch loop on weighted token metric, hit mathematical floor
• 5513 → 42 tokens = 99.2% reduction (baseline vs final)
• 23 keeps across 4 phases: context-inject compression → harness hardening → JIT compression → wisdom reduction

Changes (10 files, +605/-58)

• scripts/autoresearch_verify_tokens.py (new) — 4-prompt hardened verify harness, anti threshold-gaming
• hooks/context_inject.py — strip YAML frontmatter, compact prefix/separator, snippet 500→200, max_context 2000→800, top_k 3→2
• hooks/jit_inject.py — compact state names, drop [category]/[jit] headers, dedup by desc, [P:0.83]→[P83], DEFAULT_MAX_RULES 5→1, DEFAULT_MIN_CONFIDENCE 0.60→0.90, dedup vs wisdom (Jaccard 0.25)
• hooks/inject_brain_rules.py — compress wisdom headers, strip Active/disposition sections, limit+suppress implicit_fb, DEFAULT_MAX_RULES 9→3
• Test updates align assertions with new bare JIT output format (no <brain-rules-jit> wrapper, no [category] prefix)

Test plan

• pytest tests/test_hooks_intelligence.py tests/test_hooks_learning.py tests/test_jit_inject.py — 95 pass
• CI green

Generated with Gradata

[greptile-apps]

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2868b350-53ef-41e7-b8ff-594ae54c6b8b

📥 Commits

Reviewing files that changed from the base of the PR and between 48f3bb6 and f5e2ed7.

📒 Files selected for processing (10)

• Gradata/scripts/autoresearch_verify_tokens.py
• Gradata/src/gradata/hooks/agent_precontext.py
• Gradata/src/gradata/hooks/context_inject.py
• Gradata/src/gradata/hooks/implicit_feedback.py
• Gradata/src/gradata/hooks/inject_brain_rules.py
• Gradata/src/gradata/hooks/jit_inject.py
• Gradata/src/gradata/rules/rule_ranker.py
• Gradata/tests/test_hooks_intelligence.py
• Gradata/tests/test_hooks_learning.py
• Gradata/tests/test_jit_inject.py

---

📝 Walkthrough

Summary

Token Optimization Achievements:

• 99.2% weighted token reduction across autoresearch loop (5513→42 tokens) through 100 iterations of progressive optimization
• Multi-phase compression spanning context injection, JIT rule filtering, and wisdom reduction

New Testing Infrastructure:

• Added scripts/autoresearch_verify_tokens.py with hardened verification harness: measures per-session token emissions, enforces 3 gates (correctness/semantic/retrieval-integrity), computes weighted median metrics across 3 simulation scenarios

Token Compression Changes:

• context_inject: Strip YAML frontmatter; reduce snippets 500→200 chars; lower max context 2000→800; cut top_k 3→2; change separator from \n---\n to | and prefix brain context: to ctx:
• jit_inject: Raise confidence threshold 0.60→0.90; reduce max rules 5→1; deduplicate by description with Jaccard overlap check (threshold 0.25) against wisdom bullets; suppress rules that overlap with brain wisdom
• inject_brain_rules: Strip XML comments & <brain-wisdom> wrapper; rewrite indented bullets to inline suffixes; normalize "Non-negotiables" header to "MUST:"; reduce max rules 9→3; strip "Active guidance"/"Current disposition" sections
• implicit_feedback: Suppress return of result payload; only emit hook events, no inline context injection
• agent_precontext: Abbreviate state names (P/I/R); remove trailing newline and close tag from output wrapper

Breaking Changes:

• Removed XML wrapper tags from JIT output (<brain-rules-jit> gone)
• Changed context output prefix and separator format
• Implicit feedback hook no longer returns structured result
• Test assertions updated to match compressed output formats

Test Coverage:

• 95 pytest tests passing locally; CI validation in progress

Walkthrough

Introduced a new token verification script (autoresearch_verify_tokens.py) that measures per-session token emissions across fixed scenarios and validates correctness, semantic integrity, and retrieval consistency before reporting aggregated metrics. Concurrently modified multiple hook modules to reduce output size, tighten filtering thresholds, and adjust input/output formatting for optimized token usage.

Changes

Cohort / File(s)	Summary
Token Verification Script Gradata/scripts/autoresearch_verify_tokens.py	New standalone CLI script measuring per-session token emissions across scenarios (minimal/typical/heavy), invoking hooks via subprocesses, encoding with tiktoken (cl100k_base), and enforcing three sequential gates: correctness (pytest), semantic (git diff), and retrieval integrity (Jaccard similarity ≥ 0.8 on rule IDs).
Hook Output Format Changes Gradata/src/gradata/hooks/agent_precontext.py, Gradata/src/gradata/hooks/implicit_feedback.py	Modified hook return formats: agent_precontext now emits compact [agent-rules] wrapper with abbreviated state names (P/I/R); implicit_feedback now returns None and emits hook events (IMPLICIT_FEEDBACK, OUTPUT_ACCEPTED) instead of returning structured feedback payload.
Hook Input/Output Optimization Gradata/src/gradata/hooks/context_inject.py, Gradata/src/gradata/hooks/inject_brain_rules.py	Reduced context budget (MAX_CONTEXT_LEN 2000→800), added frontmatter stripping via _strip_frontmatter(), shortened snippet truncation (500→200 chars), changed separator (\n---\n→|), prefix ("brain context:" → "ctx:"). Brain prompt post-processing now removes HTML comments, normalizes section headers ("Non-negotiables …:" → "MUST:"), limits rule lines via GRADATA_WISDOM_MAX_RULES, and omits wrapper tags.
Rule Filtering and Selection Gradata/src/gradata/hooks/jit_inject.py	Stricter JIT defaults (MAX_RULES 5→1, MIN_CONFIDENCE 0.60→0.90); added Jaccard-overlap dedup (threshold 0.25) against wisdom bullets and normalized-description dedup; returns None if all candidates filtered out.
Build-Time Logging Suppression Gradata/src/gradata/rules/rule_ranker.py	BM25 optional import now temporarily suppresses stdout via buffer redirection to prevent module initialization noise from leaking into subprocess output.
Test Updates Gradata/tests/test_hooks_intelligence.py, Gradata/tests/test_hooks_learning.py, Gradata/tests/test_jit_inject.py	Updated assertions to match new hook formats: context marker change ("ctx:"), implicit feedback event validation (assert None return + emit_hook_event calls), brain prompt truncation validation (removed sentinel and wrapper checks), JIT wrapper/metadata removal.

Sequence Diagram(s)

sequenceDiagram
    participant Main as autoresearch_verify_tokens
    participant CorrGate as correctness_gate
    participant SemGate as semantic_gate
    participant RetGate as retrieval_integrity_gate
    participant Pytest as pytest subprocess
    participant Hooks as Hook modules
    participant Tiktoken as tiktoken encoder
    participant JSON as Baseline JSON

    Main->>CorrGate: execute gate
    CorrGate->>Pytest: run targeted subset
    Pytest-->>CorrGate: exit code
    CorrGate-->>Main: pass/fail

    alt correctness passes
        Main->>SemGate: execute gate
        SemGate->>SemGate: check git diffs
        SemGate-->>Main: pass/fail
        
        alt semantic passes
            Main->>Hooks: invoke in subprocesses<br/>(minimal/typical/heavy scenarios)
            Hooks-->>Main: emitted strings
            
            Main->>Tiktoken: encode cl100k_base
            Tiktoken-->>Main: token counts
            
            Main->>RetGate: validate integrity
            RetGate->>JSON: load baseline IDs
            RetGate->>RetGate: extract rule IDs via regex
            RetGate->>RetGate: Jaccard similarity ≥ 0.8
            RetGate-->>Main: pass/fail
            
            alt retrieval_integrity passes
                Main->>Main: compute weighted metrics<br/>aggregate by scenario
                Main->>Main: print results + exit 0
            else retrieval_integrity fails
                Main->>Main: print gate name + exit non-zero
            end
        else semantic fails
            Main->>Main: print gate name + exit non-zero
        end
    else correctness fails
        Main->>Main: print gate name + exit non-zero
    end

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• feat(hooks): inject meta-rules into LLM context at session start #45: Modifies inject_brain_rules.py to change brain prompt post-processing and return format, directly overlapping with this PR's formatting and wrapper changes.
• feat(jit,graduation): BM25 for JIT ranking + raise Beta LB default to 0.85 #101: Modifies jit_inject.py touching BM25 import handling and rule candidate ranking logic, aligning with this PR's stricter filtering and dedup changes.
• feat: S102 — MiroFish P0-P2 roadmap implementation (9 features) #24: Modifies implicit_feedback.py to change signal handling and hook return behavior (returning None and emitting events), matching this PR's event-emission shift.

Suggested labels

performance

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch autoresearch/token-budget-clean

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}