Skip to content

gate: move every objectively-checkable criterion from LLM judgment to deterministic checks #8833

Description

@JSONbored

Context

Parent: #8828 (Phase 3 — mechanism; the single largest available accuracy gain and it is not an ML technique). Most gate criteria are objectively checkable — CI state, mergeable-clean, linked-issue eligibility, path allowlist, patch coverage, generated-artifact drift. Every criterion moved from LLM judgment to a deterministic check is 100% accurate by construction and shrinks the surface the confidence/abstention machinery must cover.

Requirements

  • Inventory every input currently influencing the AI review verdict; classify each as objectively-checkable vs genuinely-subjective. Deliver the inventory as a table in the PR description.
  • For each objectively-checkable criterion still flowing through the LLM prompt: implement it as a deterministic rule (or confirm an existing rule already owns it and remove the duplicated prompt influence), so the LLM never overrides a fact.
  • The AI judge's rubric shrinks to the subjective residual, graded per-dimension with isolated judge calls (one dimension per call), each with an explicit "unknown" abstain option that maps to hold — never a guessed verdict.
  • Deterministic combination: structured per-dimension findings → verdict via a pure, tested function. No free-text parsing on the decision path.

Test Coverage Requirements

99%+ patch coverage, branch-counted; each extracted deterministic check gets both-arm tests plus a regression test proving the LLM path can no longer flip it.

Expected Outcome

The gate's accuracy ceiling rises structurally: facts are decided by code, judgment is scoped to what genuinely requires it, and abstention is a first-class output.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions