fix: harden consumer model slot resolution

[stranske]

Summary

• fail closed when a configured consumer slot allowlist is unreadable or lacks an explicit pinned model
• prevent empty placeholder models from short-circuiting provider resolution
• clarify the GitHub Models unavailable error and keep benchmark confidence bounds finite

Validation

• /opt/anaconda3/bin/python3.12 -m pytest tests/tools/test_llm_registry_selection.py tests/tools/test_llm_provider.py tests/tools/test_evaluate_model_benchmark.py tests/workflows/test_sync_manifest_delivery.py -q (119 passed)
• /opt/anaconda3/bin/python3.12 scripts/validate_template_sync.py
• git diff --check

This is source-first remediation for the active sync/workflows-7a590071b52b consumer review threads.

Summary by CodeRabbit

• Bug Fixes

  • Improved the error shown when GitHub Models can’t be initialized, with clearer install/config guidance.
  • Slot-based model selection now treats explicitly provided slot config as an allowlist and “fails closed” on unreadable/missing configs or unresolved pinned models, preventing unintended fallbacks (including when emergency bootstrap would otherwise apply).
  • Benchmark confidence-interval calculations stay finite when confidence is extremely close to 100% (avoids NaN/Inf edge effects).

• Tests

  • Added coverage for fail-closed slot config scenarios and near-100%-confidence finiteness.

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[stranske-keepalive]

Workflow source needed

PR #2785 needs either a linked GitHub issue or one valid non-issue Workflow Source before PR metadata automation can manage it safely.

Please do one of:

• Add <!-- meta:issue:123 --> or a normal Closes #123 / Related to #123 line.
• Check one Workflow Source option in the PR body.
• Add a hidden marker such as <!-- workflow-source:local_request -->, <!-- workflow-source:manual_remote -->, <!-- workflow-source:review_followup -->, <!-- workflow-source:sync_campaign -->, or <!-- workflow-source:dependabot -->.
• Add a workflow source label such as workflow:source-direct-pr, workflow:source-local-request, workflow:source-review-followup, workflow:source-sync, or workflow:no-automation.

Once a valid source is present, this warning will not be reposted.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Configured model slots now fail closed for invalid or unresolved configurations, benchmark confidence calculations clamp probabilities near their boundaries, GitHub Models errors provide specific guidance, and worker-attempt metadata is refreshed.

Changes

Model selection and evaluation safeguards

Layer / File(s)	Summary
Strict slot resolution and validation templates/consumer-repo/tools/llm_registry.py, tools/llm_registry.py, tests/tools/test_llm_registry_selection.py	Explicit slot configurations act as fail-closed allowlists; invalid, missing, or unresolved configurations return no usable slots without emergency model fallback. Tests cover these cases.
Benchmark confidence stability tools/evaluate_model_benchmark.py, tests/tools/test_evaluate_model_benchmark.py	The inverse-CDF probability is clamped away from 0 and 1, and a near-one confidence-level test verifies finite metrics.
Provider-specific client errors templates/consumer-repo/tools/llm_provider.py, tools/llm_provider.py	GitHub Models client errors provide provider-specific installation and configuration guidance.
Worker attempt metadata refresh langsmith-fleet-worker-attempt.json	The emitted timestamp and pull-request number are updated, and the existing fallback-model array is reformatted without changing its value.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• stranske/Workflows#2500: Connects provider model and client resolution to the shared model registry path.
• stranske/Workflows#2502: Handles empty model-resolution outcomes in provider and client construction.
• stranske/Workflows#2781: Overlaps with slot selection and explicit model-pin handling.

Suggested labels: maintenance, langchain, sync

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately reflects the main change: hardening consumer model slot resolution.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/sync-wave-model-registry

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/tools/test_evaluate_model_benchmark.py`:
- Around line 108-112: Update test_near_one_confidence_level_has_finite_interval
to assert finiteness of the Wilson lower- and upper-bound metrics in each
result, rather than task_success_rate. Keep the near-one confidence-level setup
and benchmark evaluation unchanged.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: a77bb2df-d80a-42a4-99de-4182dad0cd37

📥 Commits

Reviewing files that changed from the base of the PR and between 794a701 and 6a933c0.

📒 Files selected for processing (7)

• templates/consumer-repo/tools/llm_provider.py
• templates/consumer-repo/tools/llm_registry.py
• tests/tools/test_evaluate_model_benchmark.py
• tests/tools/test_llm_registry_selection.py
• tools/evaluate_model_benchmark.py
• tools/llm_provider.py
• tools/llm_registry.py

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6a933c0738

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Copilot]

Pull request overview

This PR hardens LLM model/slot resolution so consumer “slot allowlists” fail closed under invalid/unusable configurations, prevents empty placeholder models from short-circuiting provider selection, improves the GitHub Models “unavailable” error message, and keeps benchmark confidence bounds finite for extreme (but valid) confidence levels.

Changes:

• Treat malformed/unreadable slot config as an allowlist failure (fail closed) and stop substituting reviewed selections when an explicit legacy pin can’t be resolved.
• Skip empty slot models during configured_model_for_provider resolution to avoid early short-circuits.
• Clamp benchmark confidence math to avoid infinite inverse-CDF outputs near probability=1.0, and improve GitHub Models client error messaging.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tools/llm_registry.py	Tightens slot allowlist semantics and model pin handling for fail-closed resolution.
tools/llm_provider.py	Improves GitHub Models client-unavailable RuntimeError message.
tools/evaluate_model_benchmark.py	Clamps inverse-CDF probability to keep confidence bounds finite.
tests/tools/test_llm_registry_selection.py	Adds regression tests for invalid slot config and unresolved explicit pins failing closed.
tests/tools/test_evaluate_model_benchmark.py	Adds regression test for near-1.0 confidence levels producing finite metrics.
templates/consumer-repo/tools/llm_registry.py	Mirrors llm_registry allowlist hardening into the consumer template.
templates/consumer-repo/tools/llm_provider.py	Mirrors improved GitHub Models client-unavailable error message into the consumer template.

[agents-workflows-bot]

Automated Status Summary

Head SHA: dbcd0e9
Latest Runs: ⏳ pending — Gate
Required contexts: Gate / gate, Health 45 Agents Guard / guard
Required: core tests (3.12): ⏳ pending, core tests (3.13): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job	Result	Logs
(no jobs reported)	⏳ pending	—

Coverage Overview

• Coverage history entries: 1

Coverage Trend

Metric	Value
Current	75.70%
Baseline	85.00%
Delta	-9.30%
Minimum	70.00%
Status	✅ Pass

Top Coverage Hotspots (lowest coverage)

File	Coverage	Missing
scripts/issue_dedup_smoke.py	0.0%	4
scripts/runner_lib/__main__.py	0.0%	3
scripts/validate_template_sync.py	0.0%	81
scripts/langchain/topic_splitter.py	19.1%	57
tools/codex_log_analyzer.py	19.6%	140
scripts/repo_review_round2_runner.py	25.1%	344
scripts/prune_agent_stubs.py	39.7%	26
scripts/repo_review_round1_runner.py	40.7%	133
tools/ensure_workflow_timeout_variables.py	42.1%	74
scripts/sync_label_docs.py	42.9%	64
tools/discover_model_catalog.py	44.8%	55
scripts/repo_review_backlog_scan.py	45.3%	116
scripts/repo_review_body_writer.py	46.5%	86
tools/codex_session_analyzer.py	47.9%	59
scripts/create_verifier_labels.py	48.3%	58

Low Coverage Files (<50.0%)

File	Coverage	Missing
scripts/issue_dedup_smoke.py	0.0%	4
scripts/runner_lib/__main__.py	0.0%	3
scripts/validate_template_sync.py	0.0%	81
scripts/langchain/topic_splitter.py	19.1%	57
tools/codex_log_analyzer.py	19.6%	140
scripts/repo_review_round2_runner.py	25.1%	344
scripts/prune_agent_stubs.py	39.7%	26
scripts/repo_review_round1_runner.py	40.7%	133
tools/ensure_workflow_timeout_variables.py	42.1%	74
scripts/sync_label_docs.py	42.9%	64
tools/discover_model_catalog.py	44.8%	55
scripts/repo_review_backlog_scan.py	45.3%	116
scripts/repo_review_body_writer.py	46.5%	86
tools/codex_session_analyzer.py	47.9%	59
scripts/create_verifier_labels.py	48.3%	58

Updated automatically; will refresh on subsequent CI/Docker completions.

---

Keepalive checklist

Scope

No scope information available

Tasks

• No tasks defined

Acceptance criteria

• No acceptance criteria defined

[stranske]

Runner dispatch state for autofix on PR #2785. Do not edit.

[github-actions]

Runner dispatch state for codex on PR #2785. Do not edit.

[github-actions]

Autofix updated these files:

• tests/tools/test_evaluate_model_benchmark.py
• tests/tools/test_llm_registry_selection.py

[stranske]

Addressed the remaining fail-closed review findings in 016a164: explicit invalid registries and malformed UTF-8 slot configs now fail closed in both source and consumer template; added direct resolve_slots coverage and removed the Ruff regressions. Validation: ruff plus 122 focused tests and validate_template_sync all pass.