Skip to content

fix: keep template model registry fresh - #2719

Merged
stranske merged 2 commits into
mainfrom
codex/fix-template-model-registry-freshness
Jun 30, 2026
Merged

fix: keep template model registry fresh#2719
stranske merged 2 commits into
mainfrom
codex/fix-template-model-registry-freshness

Conversation

@stranske

@stranske stranske commented Jun 30, 2026

Copy link
Copy Markdown
Owner

Summary

  • sync consumer template model registry metadata with the canonical root registry
  • align the template LLM slot defaults with the non-ossifying root slot config
  • add a regression proving template defaults pass the freshness gate on 2026-06-30

Validation

  • python3 -m pytest tests/test_check_model_registry_freshness.py -q
  • python3 scripts/validate_template_sync.py
  • python3 scripts/validate_template_completeness.py
  • git diff --check
  • python3 tools/check_model_registry_freshness.py --registry templates/consumer-repo/config/model_registry.json --slots templates/consumer-repo/config/llm_slots.json --today 2026-06-30 --json

Summary by CodeRabbit

  • Bug Fixes

    • Updated the consumer template’s model and slot configuration to reflect the latest approved settings and review dates.
    • Switched one model entry to a different provider for improved consistency.
  • Tests

    • Added coverage to verify the consumer template’s registry settings remain valid and up to date.

@coderabbitai

coderabbitai Bot commented Jun 30, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: d0c1180a-d23e-4a0d-9149-6a7321f1e504

📥 Commits

Reviewing files that changed from the base of the PR and between 5afdb2a and 7ba57f5.

📒 Files selected for processing (3)
  • templates/consumer-repo/config/llm_slots.json
  • templates/consumer-repo/config/model_registry.json
  • tests/test_check_model_registry_freshness.py

📝 Walkthrough

Walkthrough

Updates the consumer-repo template configs: slot2 in llm_slots.json switches from a model name to quality_tier: "T5", codex-mini-latest provider changes from openai to github-models in model_registry.json, registry metadata dates are refreshed, and a new pytest test verifies the template defaults pass the freshness gate as of 2026-06-30.

Changes

Consumer Template Config and Freshness Validation

Layer / File(s) Summary
Template config updates
templates/consumer-repo/config/llm_slots.json, templates/consumer-repo/config/model_registry.json
slot2 switches from model: "claude-sonnet-4-6" to quality_tier: "T5"; codex-mini-latest provider changed from openai to github-models; registry last_updated, review_interval_days, and review_by updated.
Freshness gate regression test
tests/test_check_model_registry_freshness.py
New test test_consumer_template_registry_defaults_are_fresh loads the template configs and asserts gate.evaluate returns no findings for today=2026-06-30.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Possibly related PRs

  • stranske/Workflows#2709: Adds the model-registry freshness gate tool and broader unit/integration tests using the same gate.evaluate pattern against model_registry.json and llm_slots.json.
  • stranske/Workflows#2718: Overlaps on freshness gate work around quality_tier normalization and related registry/slot handling in the same files.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: refreshing the template model registry metadata.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-template-model-registry-freshness

Comment @coderabbitai help to get the list of available commands.

@stranske stranske added codex codex-automation automation Automation and workflow automation maintenance Maintenance tasks sync labels Jun 30, 2026
@stranske
stranske temporarily deployed to agent-standard June 30, 2026 05:31 — with GitHub Actions Inactive
@stranske-keepalive

Copy link
Copy Markdown
Contributor

Workflow source needed

PR #2719 needs either a linked GitHub issue or one valid non-issue Workflow Source before PR metadata automation can manage it safely.

Please do one of:

  • Add <!-- meta:issue:123 --> or a normal Closes #123 / Related to #123 line.
  • Check one Workflow Source option in the PR body.
  • Add a hidden marker such as <!-- workflow-source:local_request -->, <!-- workflow-source:manual_remote -->, <!-- workflow-source:review_followup -->, <!-- workflow-source:sync_campaign -->, or <!-- workflow-source:dependabot -->.
  • Add a workflow source label such as workflow:source-direct-pr, workflow:source-local-request, workflow:source-review-followup, workflow:source-sync, or workflow:no-automation.

Once a valid source is present, this warning will not be reposted.

@stranske
stranske enabled auto-merge (squash) June 30, 2026 05:31
@stranske
stranske temporarily deployed to agent-standard June 30, 2026 05:32 — with GitHub Actions Inactive
@stranske-keepalive

stranske-keepalive Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Automated Status Summary

Head SHA: 65e9def
Latest Runs: ⏳ pending — Gate
Required contexts: Gate / gate, Health 45 Agents Guard / guard
Required: core tests (3.12): ⏳ pending, core tests (3.13): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job Result Logs
(no jobs reported) ⏳ pending

Coverage Overview

  • Coverage history entries: 1

Coverage Trend

Metric Value
Current 75.67%
Baseline 85.00%
Delta -9.33%
Minimum 70.00%
Status ✅ Pass

Top Coverage Hotspots (lowest coverage)

File Coverage Missing
scripts/issue_dedup_smoke.py 0.0% 4
scripts/runner_lib/__main__.py 0.0% 3
scripts/validate_template_sync.py 0.0% 77
scripts/langchain/topic_splitter.py 19.1% 57
tools/codex_log_analyzer.py 19.6% 140
scripts/repo_review_round2_runner.py 25.1% 344
scripts/prune_agent_stubs.py 39.7% 26
scripts/repo_review_round1_runner.py 40.7% 133
tools/ensure_workflow_timeout_variables.py 42.1% 74
scripts/sync_label_docs.py 42.9% 64
tools/ci_failure_triage.py 44.2% 123
scripts/repo_review_backlog_scan.py 45.3% 116
scripts/repo_review_body_writer.py 46.5% 86
tools/codex_session_analyzer.py 47.9% 59
scripts/create_verifier_labels.py 48.3% 58

Low Coverage Files (<50.0%)

File Coverage Missing
scripts/issue_dedup_smoke.py 0.0% 4
scripts/runner_lib/__main__.py 0.0% 3
scripts/validate_template_sync.py 0.0% 77
scripts/langchain/topic_splitter.py 19.1% 57
tools/codex_log_analyzer.py 19.6% 140
scripts/repo_review_round2_runner.py 25.1% 344
scripts/prune_agent_stubs.py 39.7% 26
scripts/repo_review_round1_runner.py 40.7% 133
tools/ensure_workflow_timeout_variables.py 42.1% 74
scripts/sync_label_docs.py 42.9% 64
tools/ci_failure_triage.py 44.2% 123
scripts/repo_review_backlog_scan.py 45.3% 116
scripts/repo_review_body_writer.py 46.5% 86
tools/codex_session_analyzer.py 47.9% 59
scripts/create_verifier_labels.py 48.3% 58

Updated automatically; will refresh on subsequent CI/Docker completions.


Keepalive checklist

Scope

No scope information available

Tasks

  • No tasks defined

Acceptance criteria

  • No acceptance criteria defined

@stranske
stranske temporarily deployed to agent-standard June 30, 2026 05:32 — with GitHub Actions Inactive

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2abb41adb4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

"name": "slot2",
"provider": "anthropic",
"model": "claude-sonnet-4-6"
"quality_tier": "T5"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid resolving the Anthropic slot to Opus

With this template, any consumer/new repo that has an Anthropic token and uses the default slot config no longer uses the documented Sonnet compare-mode default: load_slot_config() resolves a slot that has quality_tier but no model via select_model_for_tier(), and the current registry rates claude-opus-4-6 higher on T5 than claude-sonnet-4-6 while also marking Opus very expensive. I verified the templated config loads as SlotDefinition(... model='claude-opus-4-6'), so this silently shifts verifier/compare traffic to the most expensive Anthropic model rather than just avoiding a dominated-pin freshness finding.

Useful? React with 👍 / 👎.

@stranske
stranske temporarily deployed to agent-standard June 30, 2026 07:32 — with GitHub Actions Inactive
@stranske
stranske merged commit fd8335c into main Jun 30, 2026
51 checks passed
@stranske
stranske deleted the codex/fix-template-model-registry-freshness branch June 30, 2026 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

automation Automation and workflow automation codex codex-automation maintenance Maintenance tasks sync

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants