Skip to content

orb(security): verdict-shopping — contributors can re-roll the non-deterministic AI reviewer into an auto-merge #9016

Description

@JSONbored

Security advisory (full detail + exploit): https://github.com/JSONbored/loopover/security/advisories/GHSA-5w8q-r748-9j9c

The AI verdict is re-rollable with no cross-attempt escalation, so a contributor can retry until blockers come back empty and the gate auto-merges. Threatens reward economics directly.

  • Cache is keyed on head SHA (src/queue/processors.ts ~10321-10331) → any new head is a fresh independent roll. The code acknowledges re-running "can even flip the published verdict" (src/review/ai-review-cache-input.ts ~27).
  • With a dynamic-context feature on, the cache is bypassed even at the same head, throttled only by AI_REVIEW_NON_CACHEABLE_RETRY_COOLDOWN_MS = 30min (ai-review-orchestration.ts ~99). Verified live: LOOPOVER_REVIEW_GROUNDING, _RAG, _ENRICHMENT are all true — so no commit is even needed, just CI retrigger + 30 min.
  • No cross-head escalation exists. REGATE_REPAIR_MAX_ATTEMPTS_PER_SHA=5 (~1094) is per-SHA (~3022) and resets on every commit.

Requires only fork-contributor permissions. (The maintainer-gated forceAiReview authorization is correctly enforced and is NOT the issue.)

Fix

  1. Make block-mode verdicts sticky on a content fingerprint rather than head SHA — ai-review-cache-input.ts already computes a reviewFiles patch hash, so a no-op recommit reuses the prior blocker.
  2. Persist a per-(PR, diff-fingerprint) verdict-history counter; after K disagreeing rolls, pin a manual-review hold.
  3. Raise the same-head cooldown for block mode.

Acceptance

  • A no-op recommit reuses the prior blocker instead of drawing a fresh roll.
  • N flip-flopping verdicts on one PR escalate to a human hold.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.orbGittensory Orb related - maintainer self-hosting analytics.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions