feat(standing_rules): inject_count + last_injected — named for what it measures - #296
Merged
Merged
Conversation
…t measures Deliberately NOT fired_count. A standing rule's entire lifecycle is prompt_block() -> string -> appended to sys_prompt -> sent to the LLM. Nothing downstream reports whether the model consulted the rule, obeyed it, or ignored it. Injection is the only observable event in the pipeline. Measuring real influence would need ablation (run the turn with and without the rule and diff — doubles cost, and non-determinism makes one diff weak evidence) or asking the model which rules it used, which is precisely the unverifiable self-report codec_claim_check exists to refuse. A field called fired_count would be a fabricated metric wearing an authoritative name, in the module whose whole job is refusing that. What the number is good for: inject_count == 0 PROVES dead weight — the rule never even reached the model. A high count proves nothing about usefulness, only that you're paying for the rule on every message. The listing says exactly this rather than implying the model used anything. record=True is passed ONLY by the live chat path; listing, preview and tests default to False so the metric can't measure itself. Migration is additive: existing rules keep id/text/added and gain inject_count 0 / last_injected null. The store is never reset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested as
fired_count. Shipping it asinject_count, because "fired" is not measurable here — and the requester pre-authorised that fallback if so.Where a rule is genuinely "applied"
The complete lifecycle:
That is the whole mechanism. There is no downstream point where a rule fires. Nothing in the pipeline observes whether the model consulted it, obeyed it, or ignored it.
Measuring real influence would require:
codec_claim_checkexists to refuseA field named
fired_countwould be a fabricated metric wearing an authoritative name, in the module whose entire job is refusing that.What the number is actually good for
inject_count == 0inject_countThe listing states this in the user-facing text rather than implying the model used anything, and calls out rules that have never reached the model as safe to delete.
Design details
record=Trueis passed only by the live chat path — listing, preview and tests default toFalse, so the metric cannot measure itselfid/text/addedand gaininject_count: 0/last_injected: null. The store is never reset.Verification
Migration on a real pre-existing store:
Three injections → counts
[1,1] → [2,2] → [3,3], persisted to disk with timestamps.20 standing-rules tests, including "preview does not count itself", "count survives a reload", and one asserting the listing text never uses the word "fired".