Capability Admission Provider Override Remediation

[Trecek]

Summary

Two test coverage gaps from the capability admission provider override audit:

1. test_provider_aware_capability_override_all_overridden_returns_true discards the CapabilityResolutionDetail with _ instead of capturing and asserting resolution_path == "any_pass" and bail_step is None — inconsistent with every other test in the file that exercises _provider_aware_capability_overrides with a truthy result.

2. The standalone parametrized test test_dispatch_feasible_per_backend was never added. The dispatch-feasibility assertion was embedded as a conditional block inside test_recipe_backend_matrix_cell, but the plan required a dedicated test function to isolate dispatch feasibility validation across all (recipe, backend) combinations.

The capability admission system in _provider_aware_capability_overrides uses all-or-nothing logic: it bails on the first guarded step lacking an ANTHROPIC_BASE_URL provider override, even when other guarded steps have valid overrides. This causes dispatch_infeasible for the codex backend whenever users configure only the primary implement step with a provider profile — the secondary steps (retry_worktree, fix, etc.) have no overrides, triggering partial_bail.

The architectural weakness is threefold:

1. Wrong admission semantics: The loop requires every guarded step to have a provider override, when any-suffices is correct (the runtime gate_backend_write callable provides per-step defense-in-depth).
2. Hardcoded ingredient reference: _provider_aware_capability_overrides hardcodes "inputs.backend_supports_git_write" instead of iterating CAPABILITY_INGREDIENT_TO_SKIP_GUARD, making it silently ignore future capability ingredients.
3. Stale test fixtures: Two test files maintain hardcoded _GIT_WRITE_STEPS sets that become stale when recipe steps change — and the tests that use real recipes only assert valid=True, not dispatch_feasible.

This plan provides architectural immunity by making the provider-aware override data-driven and any-suffices, replacing hardcoded step lists with dynamic discovery, and adding a structural sync test that fails immediately when the chain drifts.

Requirements

A. Diagnostics completeness in _provider_aware_capability_overrides (blast radius: LOW)

Prerequisite: applies on top of #4166 (origin/develop) — CapabilityResolutionDetail does not exist at 0.10.833.

1. Change the guarded-step resolution loop (src/autoskillit/server/tools/_auto_overrides.py) from bail-on-first-miss to full enumeration so missing_provider_steps reports ALL uncovered steps. Today the envelope names only the first uncovered step, forcing config whack-a-mole: a user covering retry_worktree is next told about the next uncovered step, one open_kitchen round trip at a time (8 total for a partial config). Verified: the three envelope branches (tools_kitchen.py, tools_recipe.py, tools_fleet_dispatch.py) read only resolution_path/missing_provider_steps — zero production consumers break; at most 3 mechanical test updates (tests/server/test_capability_admission_e2e.py:611-612,755,815) if bail_step is removed/renamed per the no-dead-fields rule; docstrings in _auto_overrides.py:62 and core/types/_type_results.py:225 go stale and must be updated.
2. Fleet fallback sentinel: in tools_fleet_dispatch.py, _capability_overrides starts {} and is populated in a try/except; on exception it stays falsy and fleet/_api.py's provider_capability_overrides or _build_capability_overrides(...) silently falls back to the non-provider-aware computation — fleet dispatches would see backend_supports_git_write=false even with a correct user config. Pre-existing defect; add an explicit sentinel or structured log.
3. Correct the record on issue _provider_aware_capability_overrides all-must-pass logic blocks Codex pipeline when only implement has provider override #4165: the [REMEDIATION] Capability Admission Diagnostics Remediation #4166 commit message claims an all→any logic change that the merged code does not implement (correctly so — any-pass was adversarially rejected as unsound). Future investigations must not believe any-pass shipped.

B. Test-seam and contract hardening

The structural hole: no test wires real ProvidersConfig → real _provider_aware_capability_overrides → real load_and_validate → open_kitchen. Server tests hardcode load_and_validate returns; the only real-recipe server test sets recipes.find=None (bypassing the provider-aware branch); recipe tests inject the capability answer directly; unit tests patch _resolve_provider_profile and use synthetic inline provider: fields real recipes don't have (Tier 3 vs the real Tiers 0–2). This is why three PRs shipped green while user-facing behavior flipped each time.

1. Seam tests (top-level functions in tests/server/test_capability_admission_e2e.py, mark medium): partial and full/wildcard config shapes — real ProvidersConfig (assert via resolved_profiles[...].raw_env["ANTHROPIC_BASE_URL"]), real recipe steps via autoskillit.recipe.io.load_recipe(...), real load_and_validate; no mocking of _resolve_provider_profile or load_and_validate. Parametrize to include remediation (guarded step names differ: assess/merge_gate_assess vs fix/merge_gate_fix).
2. Path-filter manifest fix: .autoskillit/test-filter-manifest.yaml maps src/autoskillit/recipes/*.yaml → recipe/ + contracts/ only; add server/test_capability_admission_e2e.py so recipe YAML changes run the admission tests (single-file entries are valid; the existence guard test_manifest_entry_path_exists passes).
3. Ratchet: decision-record docstring on the existing behavioral invariant test test_codex_backend_reachable_gate_returns_infeasible (tests/server/test_capability_admission_e2e.py:34) documenting the canonical contract and the 4-PR flip history, so any future assertion flip requires editing an explicit decision record. (A new arch-layer behavioral test is NOT the right home: tests/arch/test_capability_admission_control.py is small-marked/AST-structural, and duplicating the existing assertion violates redundancy rules.)
4. Config contract test (tests/config/test_providers_config.py): ANTHROPIC_BASE_URL in a profile propagates into resolved_profiles[...].raw_env — the key the entire admission path gates on, currently untested.
5. Static↔runtime key-consistency test: AST-form arch test asserting _auto_overrides.py and tools_execution.py (~line 762) gate on the same "ANTHROPIC_BASE_URL" key. A behavioral round-trip test requires extracting the inline predicate at tools_execution.py:762-767 into a helper — a production refactor to decide during planning.

C. Decision record + process

1. ADR (docs/decisions/0005-…): the codex dispatchability contract — codex + no overrides → infeasible; codex + all 9 guarded steps covered → feasible; codex + partial → infeasible with complete enumeration. Must also record the schema question any future relaxation (e.g. reachability-weighted admission) needs answered first: is review_pr.on_result[changes_requested] a happy-path or failure-path edge for admission purposes? No ADR currently answers any of this; every prior investigation re-derived the contract from the latest symptom.
2. Claimed-scope-vs-diff validation in audit-impl or review-pr: PR Remediate Missing Provider-Aware Capability Override Tests #4164 stated "All changes are test-only. No production code modifications" while modifying 5 production files (+666 lines, introducing _provider_aware_capability_overrides); the mislabel originated in issue Revert #4162: _gate_reachable_post_prune blocks working Codex+MiniMax pipeline — add provider-aware capability gate exemption #4163's framing, propagated through make-plan, and passed audit-impl (which validates diff-vs-plan, and the plan repeated the claim). Add a check that scope claims ("test-only") match changed paths.

Explicitly out of scope (evaluated and killed)

• Any-pass capability logic — unsound: uncovered guarded steps would survive pruning and run as codex workers that cannot write .git (resolve_review fires on ~75% of historical runs → mid-pipeline waste, the original [FIX] Rectify: Capability Admission Control at Recipe Load — Refuse DOA Pipelines #4094 problem). Already rejected in _provider_aware_capability_overrides all-must-pass logic blocks Codex pipeline when only implement has provider override #4165's adversarial addendum.
• Reachability-weighted admission (coverage required only for happy-path guarded steps) — HIGH breakage: the happy/recovery classification is not mechanically derivable from the recipe schema (resolve_review, merge_gate_fix, rebase_conflict_fix enter via ordinary on_result conditions indistinguishable from success routes in _build_success_step_graph); the global backend_supports_git_write ingredient unfences all guarded steps on flip; ~25 skip_when_false occurrences + contract regeneration across 3+ recipes. Blocked on the ADR question above.
• Wildcard "*" config as recommended guidance — works, but empirically reroutes 19 non-guarded steps (plan, verify, audit_impl, review_pr, diagnostics) to the override profile's model; keep the error-message example but the enumeration fix (A.1) makes surgical per-step config practical.

Closes #4167

Implementation Plan

Plan files:

• /home/talon/projects/autoskillit-runs/remediation-20260701-211618-282393/.autoskillit/temp/rectify/rectify_capability_admission_provider_override_2026-07-01_213500.md
• /home/talon/projects/autoskillit-runs/remediation-20260701-211618-282393/.autoskillit/temp/make-plan/remediation_capability_admission_provider_override_plan_2026-07-01_224500.md

🤖 Generated with Claude Code via AutoSkillit

Token Usage Summary

Step	Model	count	uncached	output	cache_read	peak_ctx	turns	cache_write	time
rectify*	opus[1m]	1	79	22.4k	3.0M	130.0k	66	137.8k	27m 18s
review_approach*	opus[1m]	1	1.5k	5.4k	222.5k	49.5k	13	34.0k	5m 6s
dry_walkthrough*	opus	2	96	17.7k	1.2M	75.2k	60	128.2k	13m 4s
implement*	MiniMax-M3	2	186.7k	23.6k	7.5M	0	142	0	8m 39s
assess*	opus[1m]	2	93	19.5k	2.4M	88.7k	80	113.8k	19m 5s
audit_impl*	opus[1m]	2	71	38.1k	820.9k	76.8k	47	113.1k	20m 49s
make_plan*	opus[1m]	1	46	5.6k	579.8k	71.3k	22	55.4k	3m 25s
prepare_pr*	MiniMax-M3	1	61.1k	3.3k	254.2k	0	16	0	50s
compose_pr*	MiniMax-M3	1	45.3k	5.7k	319.0k	0	12	0	1m 31s
review_pr*	opus[1m]	1	39	33.6k	763.0k	100.9k	33	82.4k	11m 14s
resolve_review*	opus[1m]	1	42	13.5k	2.2M	107.3k	57	88.9k	10m 48s
Total			295.0k	188.3k	19.3M	130.0k		753.8k	2h 1m

* Step used a non-Anthropic provider; caching behavior may differ.

Token Efficiency

Step	LoC Changed	cache_read/LoC	cache_write/LoC	output/LoC
rectify	0	—	—	—
review_approach	0	—	—	—
dry_walkthrough	0	—	—	—
implement	365	20659.6	0.0	64.8
assess	19	124044.4	5990.4	1025.7
audit_impl	0	—	—	—
make_plan	0	—	—	—
prepare_pr	0	—	—	—
compose_pr	0	—	—	—
review_pr	0	—	—	—
resolve_review	16	136698.2	5556.8	843.8
Total	400	48281.6	1884.4	470.9

Model Usage Breakdown

Model	steps	uncached	output	cache_read	cache_write	time
opus[1m]	7	1.8k	138.0k	10.0M	625.5k	1h 37m
opus	1	96	17.7k	1.2M	128.2k	13m 4s
MiniMax-M3	3	293.1k	32.6k	8.1M	0	11m 0s