Skip to content

chore(config): make llm_slots profile-driven so slots track reviewed selections - #870

Merged
stranske merged 1 commit into
mainfrom
chore/profile-driven-llm-slots
Jul 31, 2026
Merged

chore(config): make llm_slots profile-driven so slots track reviewed selections#870
stranske merged 1 commit into
mainfrom
chore/profile-driven-llm-slots

Conversation

@stranske

Copy link
Copy Markdown
Owner

Why

The pinned slot form does not track the model registry. tools/llm_registry.py deliberately retains an unprofiled pin while the pinned model is still lifecycle: current, so an advancing reviewed selection was silently ignored. Probed against a registry whose selections had advanced:

slot pinned resolved why
slot1 openai gpt-5.4 gpt-5.4 pin still current → retained
slot2 anthropic claude-sonnet-4-6 claude-opus-4-6 pin absent from registry → falls through to selection
slot3 github-models codex-mini-latest codex-mini-latest pin still current → retained

Only slot2 followed the reviewed selection, and only by accident of its pin being invalid.

What changed

All three slots switch to profile: verifier-balanced — the form stranske/Workflows now ships. No code change: the synced tools/llm_registry.py is already profile-aware, and this repo has no local client fork.

config/llm_slots.json is sync_mode: create_only in the Workflows manifest ("repos may customize provider preferences"), so it can only be changed here.

Verification

tools/check_model_registry_freshness.py -> 0 blocking (profile slots accepted)
resolve_slots(), current registry       -> gpt-5.4 / claude-opus-4-6 / codex-mini-latest   (unchanged today)
resolve_slots(), advanced registry      -> gpt-5.6-terra / claude-sonnet-5 / openai/gpt-5  (now tracks)
pytest tests/tools/test_llm_client_single_source.py tests/tools/test_langchain_client_config.py -> 13 passed

Order-independent w.r.t. stranske/Workflows#2852: correct against today's registry, and starts tracking the advanced selections as soon as the registry syncs.

🤖 Generated with Claude Code

…selections

The pinned slot form did not track the model registry. The shared resolver
deliberately RETAINS an unprofiled pin while the pinned model is still
lifecycle=current, so slot1 stayed on gpt-5.4 and slot3 on codex-mini-latest
even after the reviewed verifier-balanced selection advanced. Only slot2
followed the selection, and only because its pinned claude-sonnet-4-6 is absent
from the registry entirely.

Switch all three slots to `profile: verifier-balanced`, matching the form
Workflows now ships. tools/llm_registry.py is already profile-aware, so no code
change is needed here.

This file is sync_mode: create_only in the Workflows manifest, so it can only
be changed in this repo.

Verified: freshness gate 0 blocking; against the current registry the slots
resolve to today's selections (no behavior change), and against the advanced
registry they resolve to gpt-5.6-terra / claude-sonnet-5 / openai/gpt-5, so
the change is order-independent w.r.t. the registry sync. 13 LLM tests pass.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@cursor

cursor Bot commented Jul 31, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 5 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 184335d1-450d-4463-b6af-493f42e667cc

📥 Commits

Reviewing files that changed from the base of the PR and between df56bd3 and 0e20a6b.

📒 Files selected for processing (1)
  • config/llm_slots.json

Comment @coderabbitai help to get the list of available commands.

@stranske
stranske enabled auto-merge (squash) July 31, 2026 15:40
@stranske
stranske merged commit 3a57478 into main Jul 31, 2026
10 checks passed
@stranske
stranske deleted the chore/profile-driven-llm-slots branch July 31, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant