fix(review-policy): fill the sidecar catalog round-robin across credential accounts

[seonghobae]

Summary

The review sidecar (noema-review, strix, opencode-review) served a NVIDIA-only orchestrator/free catalog on 2026-09-05 although discovery had admitted free routes from three credential accounts. build_zdr_prioritized_catalog sorted eligible routes by (cost, ZDR, provider, model) and then filled the bounded catalog in that order, taking up to account_cap per account: with the sidecar's ORCHESTRATOR_CATALOG_ACCOUNT_CAP=8 and ORCHESTRATOR_CATALOG_LIMIT=12, the fill took 8 nvidia_nim + 4 nvidia_nim_sub and stopped before the alphabetically last account (openrouter) was reached. Runtime preflight then kept 2 of those 12 routes, so a stalled NVIDIA endpoint had no other account to fail over to — the noema-review 502 class tracked in contextual-orchestrator#1045.

This PR keeps the sort and fills each (cost, ZDR) tier round-robin across accounts instead. Same input now yields 4 + 4 + 4. Tier order (free before priced, ZDR before non-ZDR), account_cap, limit, and the discovery-order-independence contract are unchanged; only which rows occupy the bounded slots changes.

Evidence

noema-review run 33969842312, job 101327666732, sidecar output after ##[endgroup]:

provider secrets present: 5 of 5
using live OpenRouter ZDR endpoint feed
"free_account_diversity": 3,
"free_pool_admitted_routes": 62,
"free_selected_count": 12,        # 8 nvidia_nim + 4 nvidia_nim_sub, 0 openrouter
"probed_count": 12, "ready_count": 2, "rejected_count": 10

Why the launcher's evidence_only filter (#1476) is not the cause on the current pin: contextual_orchestrator_review_sidecar.sh:17 pins 2e414d15, which descends from contextual-orchestrator#949 (git merge-base --is-ancestor 8cd99f13 2e414d15 → yes); in that pin the OpenRouter ProviderModelSource sets no evidence_only (model_discovery.py:411-418), the only model-level evidence_only=True is opencode_go non-chat (:1398-1401), and :1667 ORs two Falses. OpenRouter rows reach the catalog builder; the selection dropped them. #1476 remains a valid hardening against a regressed pin and is complementary. Cross-referenced on #1476 (claim comment) and contextual-orchestrator#1045.

Why not lower the cap instead: with round-robin the cap no longer decides diversity (three accounts → 4/4/4 at cap 8 or 4), but with two accounts cap 8 fills all 12 slots (6/6) where cap 4 would leave four empty. #1468's purpose for the cap (no single credential owns the catalog) holds either way, so the sidecar default is untouched.

Changes

• scripts/ci/contextual_orchestrator_review_policy.py: module-level _route_tier(row, zdr_endpoints) returning (cost rank, ZDR rank), shared by the sort key; build_zdr_prioritized_catalog groups the sorted rows by tier (itertools.groupby) and fills each tier round-robin across provider_accounts until limit, honouring account_cap.
• tests/test_contextual_orchestrator_review_policy.py: test_build_catalog_interleaves_accounts_within_a_tier (8/8/8 in, limit 12, cap 8 → first three picks one per account, 4/4/4), test_build_catalog_interleaving_keeps_zdr_tier_first (an attested route still ranks above every unattested one), test_build_catalog_interleaving_skips_exhausted_accounts (5/1/2 in → a b o a o a a a).
• CHANGELOG.md: entry with the run-log numbers.

Verification

• Negative control: the three new tests were run against the unchanged selection loop first — 3 failed; after the change — passing.
• The five existing test files that call build_zdr_prioritized_catalog (review_policy, review_live_discovery_contract, bytez_catalog_integration, free_credential_admission, review_runtime_preflight): 123 passed. test_build_catalog_applies_account_cap (cap 2 → 2/2/2) and test_build_auto_catalog_order_is_independent_of_discovery_order still hold, as they must.
• Full gate on the merged tree (coverage run -m pytest tests, coverage report, interrogate): see the commit message for the exact counts.

Developer experience

One helper and one loop; the sort key reads as (*tier, provider, model) and the fill states its own invariant in a comment with the measured numbers. Priorities are still unique and descending in pick order.

User experience

A review that admits several credential accounts is served from all of them, so one stalled provider endpoint no longer turns into a held runner and a 502 to the reviewer. No configuration change is needed by any repository.

🤖 Generated with Claude Code

Summary by CodeRabbit

• 변경 사항

  • 동일한 무료/ZDR 등급 내에서 특정 계정의 경로가 먼저 소진되지 않도록, 여러 계정의 경로를 라운드로빈 방식으로 균등하게 배분합니다.
  • 계정별 경로가 고르게 선택되어 특정 제공자에 편중되는 현상을 줄이고, 카탈로그의 경로 다양성을 높였습니다.
  • ZDR 인증 경로 우선순위, 계정별 최대 수, 전체 제한 및 발견 순서와 무관한 동작은 기존과 동일하게 유지됩니다.

• 문서

  • 카탈로그 동작 변경 및 관련 운영 영향이 변경 기록에 반영되었습니다.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: e2f69d3d-98ce-48ec-8b7a-5944a07dd08e

📥 Commits

Reviewing files that changed from the base of the PR and between 7f4c5e3 and 8ec0e56.

📒 Files selected for processing (3)

• CHANGELOG.md
• scripts/ci/contextual_orchestrator_review_policy.py
• tests/test_contextual_orchestrator_review_policy.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

build_zdr_prioritized_catalog이 각 무료 및 ZDR 티어에서 계정별 라운드로빈으로 라우트를 선택하도록 변경되었습니다. 계정 cap, 카탈로그 limit, 티어 우선순위는 유지됩니다. 관련 테스트와 변경 로그가 추가되었습니다.

Changes

카탈로그 계정 분산

Layer / File(s)	Summary
티어 계산과 라운드로빈 선택 scripts/ci/contextual_orchestrator_review_policy.py	라우트 티어 계산을 _route_tier로 추출했습니다. 각 티어에서 계정 큐를 순환하며 라우트를 선택하고, cap에 도달하거나 라우트가 소진된 계정은 제외합니다.
선택 결과 검증과 변경 기록 tests/test_contextual_orchestrator_review_policy.py, CHANGELOG.md	계정 간 분산, ZDR 티어 우선, 소진 계정 처리를 검증하는 테스트를 추가했습니다. 카탈로그 동작 변경과 관련 실행 정보를 변경 로그에 기록했습니다.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 8ec0e

Catalog entries now rotate among credential accounts within each cost/ZDR tier, improving failover diversity without changing tier priority, account caps, or catalog limits. The covered selection behavior is ready to merge.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	PR 제목은 build_zdr_prioritized_catalog의 핵심 변경인 자격 증명 계정 간 라운드로빈 카탈로그 채우기를 정확하고 간결하게 설명합니다.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 2 files. (1 skipped: 1 …
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/review-catalog-account-round-robin

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[seonghobae]

Merge-order note (host 1, measured with git merge-tree --write-tree between PR heads at 17:35Z): this PR is clean against #1476 (launcher) and #1382 (sidecar ZDR evidence) — any order works. It conflicts with #1629 ("restore evidence-only review admission", which also edits scripts/ci/contextual_orchestrator_review_policy.py and tests/test_contextual_orchestrator_review_policy.py); #1629 is itself clean against main today, so whichever of the two lands second re-resolves those two files. Since this PR only touches the selection loop and adds three tests at the end of the test module, the resolution should be keep-both in both files.

[seonghobae]

Independent verification (lane:jan session) of head 8ec0e56f against an implementation I had written in parallel before your claim landed; my branch stands down and was never pushed.

Negative control reproduced. Your three new tests: 3 failed against main@7f4c5e3e's contextual_orchestrator_review_policy.py, 47 passed on this head.

Differential fuzz, 400 random discovery reports (0–14 free routes per provider across bytez, nvidia_nim, nvidia_nim_sub, openrouter; limit 1–16; cap 1–10; with and without the ZDR feed; input shuffled), comparing build_zdr_prioritized_catalog here with my independent tier-grouped round-robin:

check	result
per-account counts equal to the independent implementation	400 / 400
selected order equal to the independent implementation	400 / 400
no non-ZDR route ahead of a ZDR route	400 / 400
len(agents) == min(limit, Σ min(cap, n_account)) (no under-fill)	400 / 400
cap and limit respected; priorities unique and descending	400 / 400
selected identical for the reversed report	400 / 400

One observation, not a defect of this PR. When the report contains duplicate (provider, model) keys with different agent_ids, the reversed report selects different ids in 60 / 400 trials — and main's current fill does the same in 50 / 400, because the sort key ends at model and Python's sort is stable, so ties keep input order. Discovery does not normally emit such duplicates; if you want it fully order-independent anyway, appending str(row["agent_id"]) to the sort key does it in one line. Optional.

On the sidecar cap default: your reasoning holds — with the interleaved fill the cap no longer decides diversity, and two accounts at cap 8 fill 12 where cap 4 leaves four empty. I withdraw it from this PR's scope. The remaining wrinkle is only that _catalog_account_cap()'s docstring in the launcher names policy.DEFAULT_ACCOUNT_CAP as the single source of truth while contextual_orchestrator_review_sidecar.sh:43 restates a different one; a follow-up that makes the docstring and the sidecar agree (either value) would close that, and it does not need to ride on this PR.

---

Generated by Claude Code

[seonghobae]

Merged as f2f91b80 (squash, bypass over REST after the GraphQL budget was exhausted) — author and merger are separate sessions.

Verified on the head tree 8ec0e56f by the merging session, not relayed: merge-base == main (7f4c5e3e); 3 files (+132/−14): CHANGELOG, scripts/ci/contextual_orchestrator_review_policy.py (+53/−14), its test module (+89). Touches neither contextual_orchestrator_review_sidecar.sh (cap line kept), zdr_policy.py, nor any pinned workflow. Full gate: 2896 passed / 1 skipped, coverage 100%, interrogate 100%. The 43 ZDR/admission contract tests pass — _route_tier reuses is_zdr_model so cost/ZDR tiers are never reordered; round-robin changes which admitted routes fill the 12, not admission. Negative control: main's policy under the head's tests → 3 failed; head's policy → 47 passed.

Authorization, cited at the act from the standing directive's chicken-and-egg clause for workflow issues that create the 60-job ceiling: a scripts/ci/ review-pipeline fix for the slot burn (each gateway 502 holds a runner 180–2174 s; noema-review is the largest failing class on non-draft heads, 36 PRs at 16:52Z), structurally unable to self-validate — its own review runs main's previous policy via pull_request_target and its verdict is gated by #1929.

⑥, written before observation: count only sidecar runs created after this merge (pre-merge runs execute the old policy from their creation-time tarball). Discriminators, in order: the sidecar log's policy evidence summary selected provider distribution moves from 8 nvidia_nim + 4 nvidia_nim_sub + 0 to 4 + 4 + 4; preflight ready_count rises from 2/12; the noema-review 502 share (12 of the last 15 failures) falls as a lagging signal.

[seonghobae]

First post-merge observation (host 1), discriminator ⑥ as written before the merge. Runs on .github#1661 head 24005da6, all created 17:30:21Z — five minutes after f2f91b80 — so they vendored the new policy (vendoring contextual-orchestrator @ 2e414d15, sidecar from main).

signal	pre-merge (run 33969842312)	post-merge noema (job 101359347686)	post-merge strix (job 101359146403)
free_pool_admitted_routes / free_account_diversity	62 / 3	62 / 3	62 / 3
selected by account	8 nvidia_nim + 4 nvidia_nim_sub + 0 openrouter	4 + 4 + 4, interleaved nim, sub, or, nim, sub, or, …	4 + 4 + 4
selected zdr flags	all false	all false (case (b): same tier, round-robin)	same
preflight probed / ready / rejected	12 / 2 / 10	12 / 2 / 10	12 / 2 / 10
which routes were ready	2 × nvidia_nim	2 × openrouter (cohere/north-mini-code:free, dots-studio/dots-3-note-preview:free)	—
rejections	4 × 404 (gemma), 6 × no-status	nvidia_nim 2 × 404 + 2 × no-status; nvidia_nim_sub same; openrouter 2 × 429	—
outcome	502 to caller (gateway)	review served (served_model=dots-studio/dots-3-note-preview:free, escalations_used 3/4), then failed on local output validation (a cited line was an array index, not a source line)	scan completed (1.5 M input tokens), 3 CRITICAL findings intersecting the changed files → gate closed, exit 2

Reading: the selection change did exactly what the tests say (8+4+0 → 4+4+4, ZDR tier untouched). ready_count did not rise, but its composition flipped — every NVIDIA route was rejected at preflight this time (404s plus four no-status rejections), so without #1939 this run would have had zero ready routes and no review at all; with it, both ready routes were OpenRouter and the review actually ran. The failure class moved from "gateway 502" to "model output rejected by the validator" and "Strix produced findings", which are the failure classes a working pipeline is supposed to have.

New constraint surfaced: two of the four OpenRouter routes were rejected with 429 at preflight. OpenRouter's free tier is rate-limited per key (and per day without credits), and every sidecar boot probes its OpenRouter routes with a real completion, so under org-wide concurrent CI the preflight itself spends that budget. That is the next lever on the OpenRouter side (probe budget, 429-as-retryable at preflight, or credits) and is a separate decision from this PR. One observation each for noema and strix; the noema job has been re-run (same post-merge workflow_sha) for a second sample.

[seonghobae]

⑥ read at 21:3xZ on every completed noema-review run created after f2f91b80 (pre-merge runs execute the old sidecar from their creation-time tarball and are excluded):

run	created	branch	providers in the sidecar's policy report	preflight	outcome
33981136873	17:29:50	docs/scheduler-pre-review-hold-fol…	nvidia_nim 7 · nvidia_nim_sub 7 · openrouter 7	probed 12, ready 6	failure — HTTP Error 502 after 3122.6 s
33982955696	18:05:19	fix/codeql-rerun-missing-verdict-r…	nvidia_nim 7 · nvidia_nim_sub 7 · openrouter 7	probed 12, ready 2	success
33980923642 / 33984536425	17:25 / 18:35	(own branch / sbom automation)	log unavailable / cancelled	—	success / cancelled

Primary discriminator: confirmed. Pre-merge reports showed 8 + 4 + 0 with OpenRouter absent; post-merge OpenRouter appears at parity with each NVIDIA account, and readiness reached 6/12 on one run (was 2/12). The tier reading is consistent with the ZDR table (OpenRouter ahead or level, never behind).

Lagging signal, n=2: 1 success, 1 failure — the failure is still a gateway 502, and at 3122 s it is the longest stall observed today (pre-merge range 180–2174 s). A plausible reading is that a wider candidate set lengthens the serial failover walk before the gateway gives up, which is contextual-orchestrator#1045's mechanism (per-process breaker, no persisted state), not this PR's. Strix post-merge: 1 success / 1 failure / 1 cancelled.

Recovery for heads whose reviews ran on the old sidecar is one push merging main@f2f91b80 (a re-run keeps the old workflow_sha), as noted on this thread; doing that for the heads I own and the Autofix-assigned ones now.

[seonghobae]

One post-#1939 counter-example, with the catalog change confirmed and the stall not.

noema-review on .github#1940, run 33981136873 (created 17:29:50Z, after this merge), job 101358559717, TRUSTED_SOURCE_REF: f2f91b806122ed233e3a0e2a325246077c2e15e4 — i.e. this sidecar.

layer	observed
preflight catalog	probed_count 12, nvidia_nim 4 / nvidia_nim_sub 4 / openrouter 4 (was 8+4+0), ready_count 6, rejected_count 6
real verdict request	served_model=deepseek-ai/deepseek-v4-flash-0731, phase=response_error, duration=3122.6s, HTTP Error 502: Bad Gateway, caller attempts=1 (gateway owns repair/failover)

So the round-robin fill works — OpenRouter routes are in the pool — but the request still sat on one NVIDIA upstream for 52 minutes and came back 502 without the gateway leaving that upstream. The residual is failover on a stalled upstream (no response at all), not on a fast error, which is the CO gateway layer (ContextualWisdomLab/contextual-orchestrator#1045 / the item-4 stall), not the sidecar cap. "Fixed for every new run" should read "pool composition fixed; stall→502 still reproduces on new runs". The 57-minute slot this run held is the same slot cost as before.

[seonghobae]

Correction to my counter-example above — the gateway did fail over; I misread served_model.

From the pinned orchestrator source (peer reading, 2e414d15): the only switch condition is a per-recv socket timeout of 90 s (timeout=90, max_retries=2, backoff 0.5→8 s), with no total deadline, first-token or header timeout. Worst case per candidate ≈ 3 × 90 + (0.5 + 1 + 2) ≈ 274 s, and _invoke then walks to the next candidate. 12 candidates × 274 s ≈ 3290 s, which matches the observed duration=3122.6s — the request walked essentially the whole serving catalog, three timeouts per hop, and served_model=deepseek-ai/deepseek-v4-flash-0731 is the last candidate tried, not a stuck upstream.

So the two layers are: #1939 fixed pool composition (4/4/4 confirmed), and the residual cost is per-hop 274 s × pool size on a bad day — a wider pool makes one total failure longer, not shorter, until a first-byte/response-start timeout exists at the policy layer (an owner decision under the no-caller-wall-clock-cap policy) and per-attempt outcomes are logged at INFO and shipped with the noema/strix artifacts (today they are DEBUG-only in the sidecar stderr, which is why no job log shows attempt durations — the exact gap behind the "why 900 s / why failed" complaint). One unverified detail: whether serving uses the 6 ready routes or all 12 (6 × 274 ≈ 1640 s ≠ 3122 s would imply two passes or extra retries).