fix(verifier): vendor the LLM model registry into .workflows-lib

[stranske]

Root cause

Every verify:compare run across the fleet has been returning CONCERNS with:

Error: unverified: compare mode requires two cross-family verifier judges; available families: none.
Concerns: LLM evaluation could not run.

This has been attributed to missing/unconfigured verifier judges and escalated to the owner on at least ten sources. It is not a credentials problem. Manager-Database (and the other affected repos) already have both OPENAI_API_KEY and CLAUDE_API_STRANSKE configured, and the workflow does pass them into the compare step.

The real cause is in the vendored library. reusable-agents-verifier.yml sparse-checks-out stranske/Workflows into .workflows-lib with scripts and tools, but not config. tools/llm_registry.py resolves its configuration relative to that vendored tree:

DEFAULT_SLOT_CONFIG_PATH = Path(__file__).resolve().parent.parent / "config" / "llm_slots.json"

So the run log for the manual compare dispatch on merged Manager-Database#1474 (run 30160098574) shows:

Cannot load model registry: .../.workflows-lib/config/model_registry.json is not a file
Cannot load slot config: .../.workflows-lib/config/llm_slots.json is not a file
Skipping LLM slot without a resolved model: slot1
Skipping LLM slot without a resolved model: slot2

With no registry, every slot resolves to an empty model, build_chat_clients returns no clients, and pr_verifier.py short-circuits to the available families: none fallback before any judge is invoked.

Fix

Add config to the sparse-checkout of both workflows that vendor tools for the LangChain client:

• reusable-agents-verifier.yml — evaluate/compare judges.
• agents-verify-to-issue-v2.yml — followup_issue_generator.py, which builds the same client and had the identical omission.

With config vendored, the bundled configuration resolves two cross-family judges whose keys are already present as secrets:

SlotDefinition(name='slot1', provider='openai',    model='gpt-5.4')
SlotDefinition(name='slot2', provider='anthropic', model='claude-opus-4-6')

Test gate

tests/workflows/test_workflow_llm_installs.py:

• test_llm_workflows_vendor_the_model_registry_config — any workflow vendoring tools from stranske/Workflows must also vendor config.
• test_bundled_llm_config_resolves_two_cross_family_judges — the shipped config resolves models for at least two provider families.

Deliberate-break demonstration: reverting only the two workflow edits fails the gate with

AssertionError: agents-verify-to-issue-v2.yml vendors `tools` but not ['config']; tools/llm_registry.py
resolves config/llm_slots.json, config/model_registry.json relative to the vendored tree, so every judge
slot resolves to no model and compare mode reports "available families: none".

Restoring the edits passes. Full local run: 717 passed, 6 skipped in tests/workflows; ruff and black clean.

Impact

This is the shared root cause behind the verifier blockers currently parked for owner decision on Manager-Database#1463/#1262/#1267/#1489, Pension-Data#637, trip-planner#1493/#1495/#1496, learning-management-system#392, Workflows#2654, and Inv-Man-Intake#851. Those were all reported as "configure two cross-family judges" decisions; they should be re-dispatchable once this merges, since re-dispatch will then run against a changed configuration rather than reproducing the same missing-registry fingerprint.

Non-goals

No change to model selection policy, judge model choices, or the separate GitHub Models access gate tracked in #2768/#2819.

Summary by CodeRabbit

• Bug Fixes

  • Updated automated workflow checkouts to include required configuration files, improving reliability for verification and issue-processing workflows.
  • Improved consistency across automated checks that rely on shared tools and model settings.

• Tests

  • Added coverage to confirm workflows include all required tools and configuration.
  • Added validation that bundled language-model settings resolve providers across multiple model families.
  • Expanded checks for known verification workflows and configuration discovery.

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[agents-workflows-bot]

Workflow source needed

PR #2850 needs either a linked GitHub issue or one valid non-issue Workflow Source before PR metadata automation can manage it safely.

Please do one of:

• Add <!-- meta:issue:123 --> or a normal Closes #123 / Related to #123 line.
• Check one Workflow Source option in the PR body.
• Add a hidden marker such as <!-- workflow-source:local_request -->, <!-- workflow-source:manual_remote -->, <!-- workflow-source:review_followup -->, <!-- workflow-source:sync_campaign -->, or <!-- workflow-source:dependabot -->.
• Add a workflow source label such as workflow:source-direct-pr, workflow:source-local-request, workflow:source-review-followup, workflow:source-sync, or workflow:no-automation.

Once a valid source is present, this warning will not be reposted.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dd123ea6c5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 34 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: a0cbc976-b390-403c-8043-dc74e839858c

📥 Commits

Reviewing files that changed from the base of the PR and between 3f239a2 and 71e2b41.

📒 Files selected for processing (1)

• templates/consumer-repo/.github/workflows/agents-verify-to-new-pr.yml

📝 Walkthrough

Walkthrough

Verifier workflows now sparse-checkout the shared config directory. Tests discover LLM workflows, verify tools and config checkout paths, and validate bundled LLM configuration across multiple provider families.

Changes

LLM registry checkout

Layer / File(s)	Summary
Include shared configuration in verifier checkouts .github/workflows/agents-verify-to-issue-v2.yml, .github/workflows/reusable-agents-verifier.yml, .github/workflows/agents-verify-to-new-pr.yml	The verifier workflows add config to the shared Workflows sparse-checkout paths.
Validate registry checkout and model resolution tests/workflows/test_workflow_llm_installs.py	Tests discover LLM client workflows, require tools and config, and verify bundled configuration resolves models from at least two provider families.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• stranske/Workflows#2498: Adds shared LLM registry and configuration handling.
• stranske/Workflows#2709: Uses and validates LLM registry configuration under config.
• stranske/Workflows#2730: Updates the LLM workflow discovery and validation tests.

Suggested labels: github:actions, sync

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the primary change: vendoring the LLM model registry into .workflows-lib for verifier workflows.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch closer/verifier-vendor-llm-config

---

_{Comment @coderabbitai help to get the list of available commands.}

[agents-workflows-bot]

Automated Status Summary

Head SHA: 274d387
Latest Runs: ⏳ pending — Gate
Required contexts: Gate / gate, Health 45 Agents Guard / guard
Required: core tests (3.12): ⏳ pending, core tests (3.13): ⏳ pending, docker smoke: ⏳ pending, gate: ⏳ pending

Workflow / Job	Result	Logs
(no jobs reported)	⏳ pending	—

Coverage Overview

• Coverage history entries: 1

Coverage Trend

Metric	Value
Current	75.90%
Baseline	85.00%
Delta	-9.10%
Minimum	70.00%
Status	✅ Pass

Top Coverage Hotspots (lowest coverage)

File	Coverage	Missing
scripts/issue_dedup_smoke.py	0.0%	4
scripts/runner_lib/__main__.py	0.0%	3
scripts/validate_template_sync.py	0.0%	81
scripts/langchain/topic_splitter.py	19.1%	57
tools/codex_log_analyzer.py	19.6%	140
scripts/repo_review_round2_runner.py	25.1%	344
scripts/prune_agent_stubs.py	39.7%	26
scripts/repo_review_round1_runner.py	40.7%	133
tools/ensure_workflow_timeout_variables.py	42.1%	74
scripts/sync_label_docs.py	42.9%	64
tools/discover_model_catalog.py	44.8%	55
scripts/repo_review_backlog_scan.py	45.3%	116
scripts/repo_review_body_writer.py	46.5%	86
tools/codex_session_analyzer.py	47.9%	59
scripts/create_verifier_labels.py	48.3%	58

Low Coverage Files (<50.0%)

File	Coverage	Missing
scripts/issue_dedup_smoke.py	0.0%	4
scripts/runner_lib/__main__.py	0.0%	3
scripts/validate_template_sync.py	0.0%	81
scripts/langchain/topic_splitter.py	19.1%	57
tools/codex_log_analyzer.py	19.6%	140
scripts/repo_review_round2_runner.py	25.1%	344
scripts/prune_agent_stubs.py	39.7%	26
scripts/repo_review_round1_runner.py	40.7%	133
tools/ensure_workflow_timeout_variables.py	42.1%	74
scripts/sync_label_docs.py	42.9%	64
tools/discover_model_catalog.py	44.8%	55
scripts/repo_review_backlog_scan.py	45.3%	116
scripts/repo_review_body_writer.py	46.5%	86
tools/codex_session_analyzer.py	47.9%	59
scripts/create_verifier_labels.py	48.3%	58

Updated automatically; will refresh on subsequent CI/Docker completions.

---

Keepalive checklist

Scope

No scope information available

Tasks

• No tasks defined

Acceptance criteria

• No acceptance criteria defined

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/workflows/test_workflow_llm_installs.py`:
- Around line 284-296: Update
test_bundled_llm_config_resolves_two_cross_family_judges to accept monkeypatch,
clear ENV_SLOT_CONFIG with monkeypatch.delenv, and set DEFAULT_SLOT_CONFIG_PATH
and DEFAULT_MODEL_REGISTRY_CONFIG_PATH to the bundled LLM_CONFIG_PATHS via
monkeypatch.setattr before calling load_slot_config. Preserve the existing
file-existence and cross-family assertions.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 3721d794-02de-43b0-b508-443e1d9e09b5

📥 Commits

Reviewing files that changed from the base of the PR and between 75df03f and dd123ea.

📒 Files selected for processing (3)

• .github/workflows/agents-verify-to-issue-v2.yml
• .github/workflows/reusable-agents-verifier.yml
• tests/workflows/test_workflow_llm_installs.py

[stranske]

Opener lane (cursor) — review findings addressed; fix widened to the third verifier surface.

Two commits on top of dd123ea:

3f239a2 — the new-PR verifier path had the identical defect. agents-verify-to-new-pr.yml runs the same scripts/langchain/followup_issue_generator.py client but sparse-checked out tools without config, so verify:create-new-pr still resolved no model on either judge slot and silently took the non-LLM fallback while verify:create-issue got the fix. Added config there.

To stop this recurring, the guard no longer uses a hand-maintained parametrize list. _discover_llm_client_workflows() selects every workflow that sparse-checks-out tools from stranske/Workflows and references scripts/langchain, so any future client-vendoring workflow inherits the requirement. Discovery now yields all three: reusable-agents-verifier.yml, agents-verify-to-issue-v2.yml, agents-verify-to-new-pr.yml. test_llm_client_workflow_discovery_covers_the_known_verifier_surfaces fails if discovery ever stops seeing them, so the guard cannot silently go empty. reusable-cursor-run.yml / reusable-gemini-run.yml vendor tools only for tools/requirements.txt and never invoke the client, so they are correctly out of scope.

Also pinned test_bundled_llm_config_resolves_two_cross_family_judges to the bundled files with monkeypatch.delenv on ENV_SLOT_CONFIG/ENV_MODEL_REGISTRY_CONFIG and monkeypatch.setattr on both default paths, per the tests/tools/test_llm_registry_selection.py pattern, so an ambient override can no longer make it pass.

71e2b41 — consumer template parity. templates/consumer-repo/.github/workflows/agents-verify-to-new-pr.yml is a byte-parity copy rather than allowlisted baseline drift, so the root-only edit tripped Health 74 Template Drift on 3f239a2. Consumer repos run this workflow, so the copy needs the same entry for the fix to reach the fleet at all.

Validation. Deliberate break: removing config from the new-PR checkout fails test_llm_workflows_vendor_the_model_registry_config[agents-verify-to-new-pr.yml] with the "available families: none" explanation; restoring passes. Exact failing CI command reproduced locally — python scripts/check_template_drift.py --allowlist config/template-drift-allowlist.txt reports unallowlisted drift 0 (was 1). Full tests/workflows = 754 passed, 3 skipped. ruff and black --line-length 100 clean.

Both review threads replied and resolved. Awaiting CI on 71e2b41; the closer owns merge, then re-dispatching compare verifier on merged Manager-Database#1474 and closing #1463 on PASS.

[github-actions]

Provider Comparison Report

Provider Summary

Provider	Model	Verdict	Confidence	Summary
openai	gpt-5.6-terra	PASS	96%	The change directly fixes the sparse-checkout root cause by adding config wherever the relevant vendored LangChain/LLM tooling is checked out: the reusable verifier, verify-to-issue workflow, ver...
anthropic	claude-sonnet-5	PASS	82%	The PR directly addresses the root cause described in the issue: the sparse-checkout blocks in reusable-agents-verifier.yml, agents-verify-to-issue-v2.yml, agents-verify-to-new-pr.yml (and its temp...

📋 Full Provider Details (click to expand)

openai

• Model: gpt-5.6-terra
• Verdict: PASS
• Confidence: 96%
• Scores:
  • Correctness: 10.0/10
  • Completeness: 10.0/10
  • Quality: 9.0/10
  • Testing: 9.0/10
  • Risks: 9.0/10
• Summary: The change directly fixes the sparse-checkout root cause by adding config wherever the relevant vendored LangChain/LLM tooling is checked out: the reusable verifier, verify-to-issue workflow, verify-to-new-PR workflow, and its consumer-repository template. This makes config/model_registry.json and config/llm_slots.json available at the paths resolved by tools/llm_registry.py, allowing configured models from multiple provider families to be discovered. The added workflow-install tests provide focused coverage for sparse-checkout contents and bundled LLM configuration/model-family resolution. The implementation is minimal, readable, consistent with the existing checkout pattern, and introduces no meaningful security, compatibility, or performance risk.

anthropic

• Model: claude-sonnet-5
• Verdict: PASS
• Confidence: 82%
• Scores:
  • Correctness: 9.0/10
  • Completeness: 8.0/10
  • Quality: 8.0/10
  • Testing: 8.0/10
  • Risks: 2.0/10
• Summary: The PR directly addresses the root cause described in the issue: the sparse-checkout blocks in reusable-agents-verifier.yml, agents-verify-to-issue-v2.yml, agents-verify-to-new-pr.yml (and its template copy) were missing the 'config' directory needed by tools/llm_registry.py to resolve model_registry.json and llm_slots.json. Each affected workflow file now includes a single added 'config' line in the correct sparse-checkout list, which is a minimal, low-risk, and correct fix matching the stated root cause. The PR also extends coverage to agents-verify-to-new-pr.yml, which wasn't explicitly listed in the task list but is clearly in scope, improving completeness. A new test file adds regression tests intended to enforce that any workflow vendoring 'tools' also vendors 'config', and to validate that the bundled LLM config resolves two cross-family judges. CI reports success on the merge commit, supporting that these new tests pass. The change is narrowly scoped, readable, and carries minimal risk since it only expands what is checked out. The main residual concern is that the full test assertions could not be reviewed due to diff truncation, but given passing CI and clear correspondence between the described bug and the fix, this is a testing-visibility gap rather than a substantive concern.
• Concerns:
  • The full content of the new test file (tests/workflows/test_workflow_llm_installs.py) is truncated in the diff, so the exact assertions for the two named acceptance tests could not be fully reviewed, though helper functions for sparse-checkout parsing are visible and CI reports the Python test suite passing.
  • The fix relies on adding a single 'config' line to sparse-checkout blocks across four workflow files; if any other workflow vendors 'tools' from stranske/Workflows without config (not covered here), the same bug could recur elsewhere — the added test appears designed to catch this going forward, mitigating the risk.
  • No changes were made to config/llm_slots.json or config/model_registry.json themselves; the second acceptance criterion (two cross-family judges resolving) depends on the pre-existing config content already supporting two provider families, which is asserted by the new test but not independently verified in this diff summary.

Agreement

• Verdict: PASS (all providers)
• Correctness: scores within 1 point (avg 9.5/10, range 9.0-10.0)
• Quality: scores within 1 point (avg 8.5/10, range 8.0-9.0)
• Testing: scores within 1 point (avg 8.5/10, range 8.0-9.0)

Disagreement

Dimension	openai	anthropic
Completeness	10.0/10	8.0/10
Risks	9.0/10	2.0/10

Unique Insights

• openai: The change directly fixes the sparse-checkout root cause by adding config wherever the relevant vendored LangChain/LLM tooling is checked out: the reusable verifier, verify-to-issue workflow, verify-to-new-PR workflow, and its consumer-repository template. This makes `config/model_registry.json...
• anthropic: The full content of the new test file (tests/workflows/test_workflow_llm_installs.py) is truncated in the diff, so the exact assertions for the two named acceptance tests could not be fully reviewed, though helper functions for sparse-checkout parsing are visible and CI reports the Python test suite passing.; The fix relies on adding a single 'config' line to sparse-checkout blocks across four workflow files; if any other workflow vendors 'tools' from stranske/Workflows without config (not covered here), the same bug could recur elsewhere — the added test appears designed to catch this going forward, mitigating the risk.; No changes were made to config/llm_slots.json or config/model_registry.json themselves; the second acceptance criterion (two cross-family judges resolving) depends on the pre-existing config content already supporting two provider families, which is asserted by the new test but not independently verified in this diff summary.

🔍 LangSmith Traces

• openai
• anthropic

[stranske]

Closer verifier disposition — unanimous PASS; one concern was real and is now fixed in #2861

This PR carried verify:compare but had no verifier report for nine hours. Cause: the label was applied before the merge, and agents-verifier.yml gates on pull_request_target: labeled plus a merged PR, so the labeled event fired against an open PR and the run gated itself out. Re-dispatched manually as run 30660138895; it returned PASS from both providers (openai gpt-5.6-terra 96%, anthropic claude-sonnet-5 82%) — which is itself end-to-end proof that this PR's fix works, since compare mode previously reported available families: none on every source.

anthropic's three concerns, audited against main:

1. "The full content of tests/workflows/test_workflow_llm_installs.py is truncated, so the two named acceptance tests could not be reviewed." Review-context limitation. Both are present and passing: test_llm_client_workflow_discovery_covers_the_known_verifier_surfaces and the parametrized test_llm_workflows_vendor_the_model_registry_config.

2. "If any other workflow vendors tools without config, the same bug could recur elsewhere." Real, and it had already recurred. Commit 3f239a2 replaced the hand-maintained list with discovery, which was the right instinct, but the discovery scans only Path(".github/workflows") — while this PR also had to hand-fix templates/consumer-repo/.github/workflows/agents-verify-to-new-pr.yml. The consumer template directory this PR repaired is therefore not guarded, and two sibling templates still carried the original defect:

• templates/consumer-repo/.github/workflows/agents-issue-optimizer.yml → runs scripts/langchain/issue_optimizer.py
• templates/consumer-repo/.github/workflows/agents-80-pr-event-hub.yml → runs scripts/langchain/followup_issue_generator.py

Both reach tools/langchain_client.py, which imports load_slot_config / configured_model_for_provider from tools/llm_registry.py, resolved relative to the vendored tree exactly as this PR describes. load_model_registry() returns [] on a missing file, so the failure mode is silent degradation to no configured model. agents-80-pr-event-hub.yml is the consumer-side verifier follow-up-issue path, so it has been running unmodelled in every consumer repo.

Fixed in bounded follow-up #2861, which adds config to both templates and extends the guard's discovery over templates/consumer-repo/.github/workflows. Worth noting: after extending discovery and fixing only the issue-optimizer, the guard caught agents-80-pr-event-hub.yml by itself — a manual pass over the template directory had missed it.

3. "The second acceptance criterion (two cross-family judges) depends on pre-existing config content, asserted by the new test but not independently verified." Independently verified two ways. test_bundled_llm_config_resolves_two_cross_family_judges pins the config paths with monkeypatch so an ambient LANGCHAIN_SLOT_CONFIG / LANGCHAIN_MODEL_REGISTRY_CONFIG cannot make it pass spuriously; and live post-merge behavior confirms it, with openai + anthropic both resolving on Manager-Database #1503, #1505, #1506, #1507 and on this PR's own re-dispatch.

Disposition: PASS accepted. This PR closes no source issue, so no issue disposition applies. Remaining debt is tracked in #2861.