fix(selfhost): force JSON mode on OpenAI-compatible review calls and stop retrying deterministic-identical outputs

[JSONbored]

Summary

Half of the back-to-back "dual-model AI review did not return a usable verdict" incident fix (PRs #8735/#8737, traced 2026-07-26). Production logs showed the ollama fallback answering the review prompt with byte-identical 2,814-char markdown prose on all three attempts — structurally incapable of rescuing a failed primary, converting every primary bail into a manual-review hold. Two mechanical fixes:

Changes

• src/selfhost/ai.ts — AiRunOptions.responseFormat: "json_object"; the OpenAI-compatible chat path forwards it as response_format (Ollama's and vLLM's compat layers honor it — the model is forced into JSON mode instead of merely asked). A server that rejects the parameter with a 400 gets exactly one retry without it (degrade to the pre-change ask-nicely posture, never fail the call); non-400 errors and 400s without the declared contract throw unchanged, single fetch.
• src/services/ai-review.ts — reviews declare the JSON contract on every ai.run call (ignored by the CLIs, which already comply via prompt). The per-model retry loop stops when an attempt returns byte-identical output to the previous attempt — reviews run at temperature 0, so a repeat is deterministic and the remaining budget provably useless (same reasoning as the existing deliberate-bail/timeout/429/structural breaks; the fallback model keeps its own full budget). New identical_retry_skipped diagnostic status + ai_review_provider_identical_retry_skipped log event.

Two pre-existing tests asserting the full 3×2 retry budget used byte-identical fixtures incidentally; updated to the new per-slot 2-attempt contract with a varying-output sibling test pinning that ONLY determinism short-circuits.

Test plan

• npm run typecheck — clean
• 9 new/updated tests: response_format forwarded/omitted, 400-strip retry, format-specific 400 vs generic 400/500, identical-output early-stop (per-slot 2 attempts), varying-output full budget pinned
• All 9 ai-review* suites + selfhost-ai green (782 tests across the affected surface)
• Changed hunks: 0 uncovered statements/branches

Closes #8790

[github-actions]

Logic backtest

Replayed 0 historical case(s) for linked_issue_scope_mismatch through the base (637e43f) and head (db49fcc) versions of its detection logic (corpus checksum 4f53cda18c2b).

Backtest comparison: linked_issue_scope_mismatch

Verdict: unchanged — no comparable axis moved.

Advisory only — this check never blocks merge (#8105).

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	loopover-ui	db49fcc	Commit Preview URL Branch Preview URL	Jul 26 2026, 06:18 AM

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@637e43f). Learn more about missing BASE report.
⚠️ Report is 4 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8794   +/-   ##
=======================================
  Coverage        ?   92.26%           
=======================================
  Files           ?      797           
  Lines           ?    79589           
  Branches        ?    24118           
=======================================
  Hits            ?    73434           
  Misses          ?     5090           
  Partials        ?     1065           

Flag	Coverage Δ
backend	92.93% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/selfhost/ai.ts	98.26% <100.00%> (ø)
src/services/ai-review.ts	97.10% <100.00%> (ø)

[loopover-orb]

Tip

✅ LoopOver review result - approve/merge recommended

_{Review updated: 2026-07-26 06:30:21 UTC}

5 files · 1 AI reviewer · no blockers · readiness 100/100 · CI green · clean

✅ Suggested Action - Approve/Merge

• safe to merge

Review summary
This PR forces JSON mode via `response_format` on OpenAI-compatible chat calls with a single retry-without-the-param on a 400 (falling back to the pre-existing ask-nicely posture), and adds a per-model identical-output short-circuit to the review retry loop so a deterministic (temp-0) repeat doesn't burn the remaining retry budget. I traced both mechanisms end-to-end: `chatBody(withResponseFormat)` in src/selfhost/ai.ts:394 only injects the field when the caller declared the contract, the 400-retry path is gated correctly on `res.status === 400 && options.responseFormat === 'json_object'`, and in ai-review.ts the `lastRawText` comparison only fires on `attempt > 0` with non-empty text, matching the updated 2-attempt-per-slot test expectations (6→4 calls). The updated fixtures correctly distinguish identical vs varying output, so the tests aren't fabricated against a scenario the code can't hit.

Nits — 6 non-blocking

• src/selfhost/ai.ts:395 — the 400-retry fires on ANY 400 while `responseFormat` is declared, not specifically a response_format-rejection 400; an unrelated 400 (bad model, auth) costs one wasted extra round trip before the real error surfaces, harmless but imprecise given the PR description calls this 'format-specific'.
• The external brief's 'debug leftover' flag on src/services/ai-review.ts:1249's `console.warn` is a false positive — it's a structured `JSON.stringify` log line consistent with the file's existing `ai_review_missing_assessment_exhausted`-style warn/error logging convention, not leftover debug output.
• src/selfhost/ai.ts and src/services/ai-review.ts are both already over the repo's 400-line file-size heuristic pre-existing this diff; not something to block on here, but worth a follow-up split if either grows further.
• Consider checking the 400 response body for a response_format-specific rejection message before retrying without it, to avoid the wasted round trip on unrelated 400s.
• The identical-output skip could log which attempt's text matched (not just the current attempt) for easier production debugging, though the current diagnostic is already sufficient to reproduce the fix(selfhost): enforce structured JSON output for OpenAI-compatible review calls, and stop retrying a deterministic-identical response #8790 incident shape.
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.

Decision drivers

• ✅ Code review — No blockers (1 reviewer)
• ✅ Gate result — Passing (No configured blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#8790
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 13 registered-repo PR(s), 13 merged, 275 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 13 PR(s), 275 issue(s).
Improvement	✅ Minor	risk: clean · value: minor · LLM: moderate

Linked issue satisfaction

Addressed
The diff adds response_format:"json_object" forwarding on the OpenAI-compatible chat call with a graceful 400-only retry fallback, implements a byte-identical-output early-stop per model with a distinct diagnostic/log event, and includes tests covering both mechanisms plus updated pre-existing tests for the new retry contract.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: not available
• Official Gittensor activity: 13 PR(s), 275 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Start here: Treat this as maintainer-lane context rather than normal contributor-lane activity.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review

[loopover-orb]

LoopOver approves — the gate is satisfied and CI is green.