feat(ci): vendored-config schema guard (A3)

[mrobinson2]

What

A new validate-vendored-config CI job that validates every config key AAF ships to a vendored app (compose env blocks, Terraform env blocks, the generated Hermes config.yaml) against the schema the pinned vendored source actually reads. A key the app silently drops — the exact class behind the Honcho 3.0.7 flat-key incident ("boots clean, runs every specialist on direct-OpenAI gpt-5.4-mini") and the Hermes config-schema-shift incident ("provider=openrouter, model=empty, 401") — now fails the build instead of silently degrading a deployment.

Design doc first, as specified: docs/design/vendored-config-schema-guard.md.

REAL DRIFT FOUND (the headline)

The guard's first honest run found the Honcho 3.0.7 incident alive in this repo — the vendor bump that migrated docker-compose.yml and .env.example (#111) missed two artifacts:

1. infrastructure/modules/container-apps/honcho.tf — all three resources (API app, always-on deriver, scheduled deriver job) still shipped the pre-3.0.7 flat keys (SUMMARY_PROVIDER, SUMMARY_MODEL, DERIVER_PROVIDER, DERIVER_MODEL, DIALECTIC_LEVELS__<lvl>__PROVIDER/__MODEL/__THINKING_BUDGET_TOKENS × 5 levels) — 57 removed env keys, all silently ignored (extra="ignore"). A Terraform deploy of the pinned Honcho 3.0.11 image would have run every specialist and all five dialectic levels on direct-OpenAI gpt-5.4-mini. The file's own comment even warns about "the old DIALECTIC_MINIMAL_PROVIDER style is silently ignored" — while using the other silently-ignored style. Fixed: migrated to the nested MODEL_CONFIG shape (values preserved; MAX_TOOL_ITERATIONS survived 3.0.7 and stays a per-level field).
2. deploy/mac-site/docker-compose.yml — same flat keys in the honcho service (19 keys), three lines below a comment correctly stating "the flat form is ignored". Fixed the same way.
3. infrastructure/modules/container-apps/hermes.tf — HERMES_DB_PATH appears nowhere in the pinned Hermes (v2026.5.16): hermes_state.py has no path override, so the documented SQLite-on-SMB mitigation for state.db was silently inert (state.db actually lives on the Azure File Share). Removed with an honest note; the twin HERMES_KANBAN_DB fix IS read (hermes_cli/kanban_db.py) and stays. Also removed the unread LOG_LEVEL from the hermes container (the model-router sidecar reads its own and keeps it). .env.example's dead HERMES_DB_PATH knob dropped too.
   Observation, not changed here: apps/paperclip/patch-adapter.mjs sets a per-session HERMES_DB_PATH for the same reason — equally a no-op on this pin; worth revisiting at the next Hermes bump.

Per-app validation strategy

App	Strategy
Honcho	Load the vendored pydantic-settings model (apps/honcho/src/src/config.py, standalone import) and derive the full accepted env-key universe: per-section env_prefix, __ nested paths through sub-models (incl. …MODEL_CONFIG__OVERRIDES__BASE_URL), dict[Literal[…]] closes the dialectic level-name set (a typo'd level is flagged), aliases honored, case-insensitive. Zero curation — a vendor bump recomputes the universe and fails iff shipped keys became invalid. A curated removed-keys map upgrades diagnostics to migration hints (SUMMARY_PROVIDER → "REMOVED in 3.0.7 — replace with SUMMARY_MODEL_CONFIG__TRANSPORT").
Hermes config.yaml	AST-parse the vendored parser (hermes_cli/config.py DEFAULT_CONFIG tree ∪ _KNOWN_ROOT_KEYS; no imports, no heavy deps) and validate the entrypoint-heredoc-generated config. Polymorphic sections (dict-form model: whose vendored default is a scalar) come from scripts/vendored-config/manifest-hermes.yaml, pinned to the submodule gitlink — a Hermes bump fails the job until the manifest is re-validated. That loud failure is the point.
Hermes env	Static consumption scan (os.getenv/os.environ literals across the vendored tree + the parser's env tables + $VAR reads in bundled AAF override/agent-runtime scripts). Over-accepts by design — its job is catching keys the app never reads. Terraform env is scoped per-container so the model-router sidecar isn't validated against Hermes.
PaperClip	Source is cloned at image build (not in-tree), so a curated manifest (manifest-paperclip.yaml) carries only upstream-consumed keys, pinned to PAPERCLIP_VERSION and cross-checked against both the Dockerfile ARG and the compose default — a version bump fails until re-validated. Everything in-tree (entrypoint, auth-proxy.mjs, patches, bundled Hermes CLI) is derived by scanning, not curated.

Failure policy: every finding is a hard fail — no warn tier (warnings in green builds are how the incidents shipped). Suppression only via scripts/vendored-config/allowlist.yaml, where an entry without a non-empty reason is itself a failure. Current allowlist: platform env consumed by the runtime, not the app (AZURE_CLIENT_ID → azure-identity, APPLICATIONINSIGHTS_CONNECTION_STRING → App Insights, GOOGLE_WORKSPACE_CLI_CREDENTIALS_FILE → the gws CLI binary).

Test evidence

Local run (Python 3.13, pydantic 2.13.4 / pydantic-settings 2.14.2):

$ python scripts/validate_vendored_config.py          # before fixes: 78 findings
vendored-config schema guard: 78 finding(s)           # 76 honcho removed-key + 2 dead hermes keys
$ python scripts/validate_vendored_config.py          # after fixes
vendored-config schema guard: clean — every shipped key is read by the pinned vendored source (or explicitly allowlisted with a reason)
$ python scripts/validate_vendored_config.py --self-test
self-test OK: seeded removed/unknown/typo'd keys all detected; valid keys and allowlisted keys pass
$ pytest -q tests/vendored-config
19 passed in 4.22s

Unit tests cover: removed-flat-key detected with migration hint; unknown key detected per app; typo'd dialectic level name detected (closed Literal set); valid nested/OVERRIDES/case shapes pass; allowlisted key passes; reason-less allowlist entry rejected; both manifest version-pin tripwires; heredoc-generated configs validate; per-container tf scoping excludes the sidecar; whole-repo clean run; self-test passes. terraform fmt/validate and docker compose config -q still green.

Vendor-bump behavior (by design)

• Honcho: universe recomputed from the new pin → fails iff shipped keys became invalid (that IS the incident, caught).
• Hermes: manifest pinned_commit ≠ new gitlink → fails always until the polymorphic sections are re-validated.
• PaperClip: manifest pinned_version ≠ Dockerfile/compose pin → fails always until the env list is re-validated.

🤖 Generated with Claude Code

[mrobinson2]

Rebased onto main after #115 (A1) and #116 (A6) landed. The new job's first CI run actually caught the A1 collision: A1 moved the Hermes config.yaml generation out of apps/paperclip/docker-entrypoint.sh into write-hermes-config.sh, and the guard failed loudly on the merge commit with "expected a HERMES_EOF heredoc … found none" — the exact moved-generation tripwire working as designed. The generator targets are now discovered (scan apps/paperclip/*.sh + services/paperclip/*.sh for the heredoc marker; zero generators = a finding) instead of hardcoded, and A1's new model.api_key / api_mode: chat_completions shape validates against the pinned Hermes schema. All local checks green again: guard clean, self-test OK, 19/19 pytest.