feat(shadow): add bounded deterministic evidence (#150)

[tcballard]

Summary

Implements issue #150, “Enterprise evidence: add deterministic shadow routing,” as the next stacked change on PR #163.

Adds:

• an opt-in, deterministic shadow evaluator over existing named route presets;
• bounded prompt-free records with scorer, configuration, candidate, status, and reason-code fingerprints;
• separate semaphore, retention, sampling, provider-comparison consent, and provider-budget controls;
• bounded Prometheus coverage, divergence, error, and explicitly authorized comparison-cost metrics;
• cooperative stop/resume at the hot-reload configuration boundary;
• configuration validation/round-trip tests, shadow-store tests, metrics tests, and the accepted WF-ADR-0063/roadmap/changelog updates.

Scope

Included

• [gateway.shadow] parsing, validation, rendering, and disabled compatibility defaults.
• Off-path decision evaluation that reuses the authoritative deterministic score and hard destination eligibility.
• Named-route counterfactual candidate selection with bounded process-local evidence.
• Double-opt-in provider comparisons requiring configuration, x-wayfinder-shadow-provider-consent, and hosted-allowed privacy posture.
• CLI wiring, runtime-state continuity across reloads, cooperative stop behavior, and bounded metrics.

Excluded

• Evidence reports, statistical sufficiency gates, or fleet aggregation (issue Enterprise evidence: generate quality and efficiency reports #151).
• Canary assignment, rollout tripwires, or enforcement controls (issue Enterprise evidence: add canary rollout and quality tripwires #152).
• OpenAI Responses API compatibility (issue Gateway compatibility: implement the OpenAI Responses API #153).
• Prompt/response retention, hidden reasoning capture, model-based judging, or any change to the production routing decision.

Product / Architecture Decisions

• Shadow work is observational and never participates in production admission, budget, retry, reliability, cache, or provider-health state.
• Candidate routes are existing named presets, validated at config load; the production scored decision remains unchanged.
• Sampling is deterministic from request ID plus a versioned candidate fingerprint.
• Records remain prompt-free and bounded. Provider comparisons are separately budgeted and require explicit request consent in addition to configuration.
• Hot reload retains the process-lifetime bounded store but flips an active flag, so disabling the policy cooperatively stops queued and in-flight work.
• Existing deployments remain no-ops because the default shadow policy is disabled.

User-Facing Contract

CLI

No new command is added. Configure the gateway:

[gateway.routes.on-device-first]
models = ["apple-local", "cloud"]

[gateway.shadow]
enabled = true
sample_rate = 0.05
candidate_routes = ["on-device-first"]
max_in_flight = 2
max_records = 2048

Provider comparison requires both provider_comparisons = true and a request header x-wayfinder-shadow-provider-consent: true; it is considered only for hosted-allowed requests.

Human Output

No prompt, response, credential, or prompt-derived identifier is emitted by the shadow evaluator. Production HTTP responses and failures are unchanged by shadow work.

JSON Output

The bounded AppState::shadow_snapshot contract exposes prompt-free records for the following evidence slice; report serialization and operator exposure are intentionally deferred to issue #151.

Exit Codes

• 0: Existing gateway startup and request behavior.
• 1: Existing runtime/provider failure semantics.
• 2: Existing configuration validation/startup failure semantics.

Verification

Ran

• cargo fmt --all -- --check
• cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
• cargo test --workspace --all-targets --locked

Covered

• Shadow policy round-trip and invalid/unknown candidate rejection.
• Deterministic sampling/fingerprints and active stop/resume behavior.
• Bounded event/cost metric rendering and label safety.
• Full Rust workspace parity, gateway, provider, CLI, Redis-live, and existing 226 gateway unit tests; all passed.

Review Path

Suggested review order: wayfinder-config shadow schema and validation; wayfinder-gateway/src/shadow.rs evaluator/store and privacy gates; AppState scheduling/reload wiring; metrics; ADR/roadmap/changelog; tests.

Notes For Reviewer

This PR targets codex/enterprise-deployment-selection (PR #163) and is intentionally draft/stacked. The retained record store is process-local and prompt-free; issue #151 will add the versioned evidence report/operator surface before any canary work. Provider comparison is deliberately not on by default and never runs for local-only privacy posture.