feat(llm): model-registry freshness gate (maint-77) — stop old models ossifying as primary - #2709
Merged
Merged
Conversation
Detects when the canonical LLM model config has gone stale so old models do not get stuck as the primary ones indefinitely. Offline/stdlib-only check over config/model_registry.json + config/llm_slots.json; flags review_overdue, blocked_pin, unknown_pin, and dominated_pin (a pin the registry's own quality scores already rate below a newer same-provider model). - tools/check_model_registry_freshness.py (+ tests, 10 cases) - config/model_registry.json: add review_interval_days + review_by - .github/workflows/maint-77-*: weekly schedule + PR gate; opens/refreshes one tracking issue on staleness (no provider API calls, no secrets) - sync-manifest: ship the gate to consumers (copy) Does not change model selection and does not touch maint-68's sync core; validate_template_sync.py still passes. On real config today it correctly flags the registry review overdue + slot2 (sonnet-4-6) dominated by opus-4-6. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
📝 WalkthroughWalkthroughAdds an offline CLI freshness check for the model registry and slot config, a new GitHub Actions workflow to run it on schedule, manual dispatch, and relevant PRs, plus registry metadata updates, tests, sync-manifest registration, and workflow docs. ChangesModel Registry Freshness Gate
Sequence Diagram(s)sequenceDiagram
participant GitHubActions as GitHub Actions
participant CheckTool as check_model_registry_freshness.py
participant GitHubCLI as gh
participant IssueTracker as Tracking Issue
GitHubActions->>CheckTool: run freshness check on schedule/PR/dispatch
CheckTool-->>GitHubActions: findings JSON and exit code
GitHubActions->>GitHubCLI: search/create/update tracking issue
GitHubCLI->>IssueTracker: comment or create issue
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related issues
Suggested labels
🚥 Pre-merge checks | ✅ 2 | ❌ 3❌ Failed checks (3 warnings)
✅ Passed checks (2 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
stranske
temporarily deployed
to
agent-high-privilege
June 29, 2026 03:28 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This was referenced Jun 29, 2026
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a1d6364797
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/maint-77-model-registry-freshness.yml:
- Around line 45-50: The freshness gate step in the workflow is treating every
non-zero result from check_model_registry_freshness.py as staleness, but rc=2
must remain a hard error. Update the logic around the gate/handling steps so the
workflow only follows the stale-finding path when the gate exits 1, and fails
the job immediately for 2 or any other unexpected exit code; use the existing
gate id and the freshness.json handling to preserve the checker’s exit-code
contract.
- Around line 23-25: The freshness-check job is exposing a write-scoped workflow
token while running PR-controlled code, and it also treats every nonzero exit as
stale. Update the job that runs `actions/checkout` and
`tools/check_model_registry_freshness.py` to disable persisted checkout
credentials, keep the `issues: write` permission off the PR-executed path, and
adjust the exit-code handling so only `1` is reported as “stale” while `2` is
handled separately.
In `@tools/check_model_registry_freshness.py`:
- Around line 197-198: The CLI path in check_model_registry_freshness.main
should treat an invalid --today as a config/usage error instead of letting
_parse_date(args.today) raise a traceback. Wrap the today parsing in the same
error-handling path used for other CLI/config validation, catch the ValueError
from _parse_date, and exit with the documented code 2 before calling evaluate.
- Around line 146-160: The dominated_pin check in
check_model_registry_freshness.py is comparing models using headline quality
across all tiers, which can flag valid tier-specific pins as dominated. Update
the logic around the pinned_q/better computation to use slot["quality_tier"]
when it exists, and only fall back to _headline_quality for pins without a tier.
Make sure the comparison against by_provider, _headline_quality, and the
generated findings message all reflect the slot’s requested tier so
dominated_pin only reports true regressions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: b18663c3-963a-4561-83cd-e475ad5852e4
📒 Files selected for processing (5)
.github/sync-manifest.yml.github/workflows/maint-77-model-registry-freshness.ymlconfig/model_registry.jsontests/test_check_model_registry_freshness.pytools/check_model_registry_freshness.py
Contributor
Automated Status SummaryHead SHA: accfd30
Coverage Overview
Coverage Trend
Top Coverage Hotspots (lowest coverage)
Low Coverage Files (<50.0%)
Updated automatically; will refresh on subsequent CI/Docker completions. Keepalive checklistScope
Verified 2026-06-28 (against local clones + this repo's
So the canonical synced files and the consumer copies have diverged, with the consumer version being the more capable one (registry + blocked-model guards) and the dependency it needs ( Tasks
Acceptance criteria
|
Contributor
|
Runner dispatch state for codex on PR #2709. Do not edit. |
stranske
temporarily deployed
to
agent-high-privilege
June 29, 2026 03:40 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
stranske
temporarily deployed
to
agent-high-privilege
June 29, 2026 20:28 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Owner
Author
|
Closer pass pushed Changes made:
Validation run locally from a disposable
Post-push readback: PR head is |
Owner
Author
|
Follow-up push
Local validation after this commit:
Post-push readback: head |
stranske
temporarily deployed
to
agent-high-privilege
June 29, 2026 20:32 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/workflows/maint-77-model-registry-freshness.yml (1)
38-46: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick winPin both external actions to commit SHAs in both jobs.
actions/checkout@v7andactions/setup-python@v6are still mutable tags here, so an upstream retag or compromise could change behavior across synced repos. Lines 38-46 and 83-91.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/maint-77-model-registry-freshness.yml around lines 38 - 46, Pin the external GitHub Actions used in both jobs to immutable commit SHAs instead of version tags: replace the uses of actions/checkout and actions/setup-python in the workflow with their corresponding commit-pinned references. Update both the first job and the second job so the Checkout and Setup Python steps are fixed to SHAs, keeping the same inputs like persist-credentials and python-version unchanged.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/ci/WORKFLOWS.md`:
- Line 205: Update the workflow description for
maint-77-model-registry-freshness.yml so it also mentions the PR trigger for
changes to tools/check_model_registry_freshness.py. The current text in the
docs/ci/WORKFLOWS.md entry understates the trigger set by only calling out
registry/slot changes; revise that sentence so it reflects both PRs for
registry/slot changes and PRs that modify the checker path, while keeping the
rest of the checker summary intact.
---
Outside diff comments:
In @.github/workflows/maint-77-model-registry-freshness.yml:
- Around line 38-46: Pin the external GitHub Actions used in both jobs to
immutable commit SHAs instead of version tags: replace the uses of
actions/checkout and actions/setup-python in the workflow with their
corresponding commit-pinned references. Update both the first job and the second
job so the Checkout and Setup Python steps are fixed to SHAs, keeping the same
inputs like persist-credentials and python-version unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 0e934599-5b9a-4906-8ff5-282476fe5ba9
📒 Files selected for processing (6)
.github/workflows/maint-77-model-registry-freshness.ymldocs/ci/WORKFLOWS.mddocs/ci/WORKFLOW_SYSTEM.mdtests/test_check_model_registry_freshness.pytests/workflows/test_workflow_naming.pytools/check_model_registry_freshness.py
stranske
temporarily deployed
to
agent-high-privilege
June 29, 2026 21:25 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Owner
Author
|
Closer follow-up pushed two commits on
Validation run locally with Python 3.12 tooling:
Post-push readback: head is |
stranske
temporarily deployed
to
agent-high-privilege
June 29, 2026 21:26 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
stranske
temporarily deployed
to
agent-high-privilege
June 29, 2026 22:23 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Contributor
Provider Comparison ReportProvider Summary
📋 Full Provider Details (click to expand)openai
anthropic
Agreement
Disagreement
Unique Insights
🔍 LangSmith Traces |
Owner
Author
|
Closer verifier disposition for the 2026-06-29 provider comparison report: treating the non-PASS result as a false positive against current Evidence checked on live
The verifier appears to have judged the stale original issue text/diff summary instead of the post-merge live-main state and the issue-owner disposition. No bounded code follow-up is required for #2709. |
This was referenced Jun 30, 2026
9 tasks
This was referenced Jul 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2708
Automated Status Summary
Scope
tools/langchain_client.pyandtools/llm_provider.pyare sync-managed from this repo to consumers (.github/sync-manifest.yml:670and:667). Two consumers have forked these synced files to route model selection through a localtools/llm_registry.pythat does not exist in Workflows and is not in the sync-manifest — so the nextmaint-68sync will overwrite the consumer forks and orphan theirllm_registry.py, silently reverting registry-driven model selection and blocked-model guards fleet-wide.Verified 2026-06-28 (against local clones + this repo's
main):tools/langchain_client.py:0references tollm_registry; slot logic is inlined (SlotDefinition(... model="gpt-5.4")atlangchain_client.py:98,build_chat_clientat:204).tools/llm_provider.pyhardcodesgpt-5.1-codex(:594,629) andclaude-sonnet-4-5-20250929(:660,705).Manager-DatabaseandTrend_Model_Project:langchain_client.pyhas 6llm_registryreferences each; both shiptools/llm_registry.pythat is byte-identical (md510527ff44035e70636dbfa9738433895) — a shared, registry-driven slot/model-selection module with blocked-model guards. (Travel-Plan-Permission carries only an emptyllm_registry.pyplaceholder; not forked.)tools/llm_registry.pyis absent from this repo (find→ none) and has 0 entries in.github/sync-manifest.yml.So the canonical synced files and the consumer copies have diverged, with the consumer version being the more capable one (registry + blocked-model guards) and the dependency it needs (
llm_registry.py) living only in consumers and outside the sync system.Tasks
main, 2026-06-28; consumer clones at theirorigin/main. Manifest evidence:.github/sync-manifest.yml:667(llm_provider.py),:670(langchain_client.py),:673(embedding_provider.py). Cross-fleet footprint: forked instranske/Manager-Databaseandstranske/Trend_Model_Project(identicalllm_registry.py). Surfaced by the Manager-Database full audit (Code/Audits/Manager-Database/2026-06-28-03-sync-drift.md). Related smaller drift in the same audit:tools/requirements-llm.txt(template_sync: exact, consumer pins ahead) and opposite-direction drift inscripts/langchain/followup_issue_generator.pyvsprogress_reviewer.py(build_chat_clientvs_llm_client.build_client) — worth folding into the same reconciliation.Acceptance criteria
state_fingerprint/ sync-drift check showstools/langchain_client.py,tools/llm_provider.py, andtools/llm_registry.pyCLEAN (identical) between this repo and Manager-Database + Trend_Model_Project — i.e. no consumer fork remains.tools/llm_registry.pyis listed in.github/sync-manifest.yml(Option A) OR is absent from all consumers (Option B).build_chat_clientselects the configured slot model rather than a hardcoded default.llm_registrymanifest entry (Option A) makes the sync-drift check flaglangchain_client.pyas DRIFT against the two consumers; restoring it returns CLEAN.Head SHA: 55750bd
Latest Runs: ✅ success — Gate
Required: gate: ✅ success
Summary by CodeRabbit
review_interval_daysandreview_bysettings to the model registry.