Skip to content

ai(cache): review fingerprint omits prompt and verdict-logic version — superseded verdicts replay forever (11 unbumped changes) #9477

Description

@JSONbored

Summary

getCachedAiReview replays a cacheable=1 row unboundedly (there is no maxAgeMs), and the cache fingerprint does not include the reviewer prompt, the verdict-combination logic version, or several per-repo flags that change what the model is asked. So a verdict produced under superseded policy is replayed verbatim — including a severity: "critical" ai_consensus_defect — and the fixed code never runs for that head.

Mechanism (verified at HEAD, 776c414)

Also absent from the fingerprint: findingCategories and improvementSignal (both append system-prompt suffixes — :1268, :1278), screenshotEvidenceSummary (appended to the system prompt, :1320), AI_MAX_OUTPUT_TOKENS, and the operator plan's onMerge (reviewerPlan hashes only combine and reviewers).

Concrete failure

A PR is closed or held under the pre-#9074 "false consensus" logic. #9074 lands, requiring genuine agreement. The PR is re-gated at the same head — identical fingerprint, cache hit — so the old critical finding and its close rationale are replayed and the corrected logic never executes. The fix is invisible for every already-reviewed head.

Requirements

  1. Anything that changes what the model is asked, or how its answer is judged, must change the cache key.
  2. Correctness must not depend on a human remembering to bump a constant — that mechanism has already failed 11 times.

Deliverables

  • Fold REVIEW_PROMPT_VERSION and sha256(buildCanonicalJudgePrompt()) into the cache payload. Both are pure and available before the call, so prompt drift becomes an automatic miss rather than a manual bump.
  • Add the missing per-repo/plan inputs: findingCategories, improvementSignal, screenshotEvidenceSummary, AI_MAX_OUTPUT_TOKENS, onMerge.
  • Bump AI_REVIEW_CACHE_INPUT_VERSION once as part of this change, to invalidate rows written under the incomplete fingerprint.
  • Consider a maxAgeMs backstop on durable reuse, so a fingerprint gap can never mean "forever".
  • Add a drift check: fail CI when a file that contributes to the prompt changes without the digest mechanism covering it — or, better, make the digest structural so the check is unnecessary.

Tests (must fail against current main)

  • Same head, prompt text changed ⇒ cache miss.
  • Same head, findingCategories / improvementSignal / onMerge toggled ⇒ cache miss for each.
  • Same head, nothing changed ⇒ cache hit (no regression in hit rate for the genuine case).
  • A row written under v6 is not served after the bump.

Expected outcome

A policy or prompt fix takes effect on the next pass for every PR, instead of being masked by a cache row that predates it.

Metadata

Metadata

Assignees

Labels

gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions