feat(api): SP-5 PR-C · /metrics + structured JSON logging with secret scrubbing (AIN-238 + AIN-249)

[hizrianraz]

Summary

Internal-scoped Prometheus `/metrics` + a structured JSON log formatter that scrubs secrets before bytes leave the process.

Stacks on SP-2 api#72. Independent of SP-5 PR-A (#76) and SP-5 PR-B (#77).

AIN-238 — /metrics

• Dependency-free registry (`services/metrics.py`); we don't pull `prometheus_client` because every runtime dep is an AIN-237 attack surface
• Named series: `ainfera_http_requests_total`, `ainfera_http_request_duration_seconds`, `ainfera_provider_calls_total`, `ainfera_router_alias_hit_total`, `ainfera_audit_chain_height` + `freshness_seconds`, `ainfera_dispatch_without_capture_total` (SP-4 PR-A bridge), `ainfera_cost_killswitch*` (SP-5 PR-B bridge), `ainfera_app_info{version}`
• Middleware times every request; uses FastAPI route template for the path label so agent_id never leaks
• `GET /metrics` gated by `X-Ainfera-Internal-Key`; hidden from openapi
• Cold-path reads `max(seq)` + `max(created_at)` from audit_events — read-only, never mutates the chain

AIN-238 — structured JSON logging

• `StructuredJSONFormatter` — one JSON object per record + 2-layer secret scrubbing (per-key for extra fields; regex for freeform text shapes `ai_infera_` / `sk-` / `Bearer` / JWT)
• Tracebacks also scrubbed
• Wired via `logging.basicConfig(force=True)` BEFORE routers import so startup lines flow through

AIN-249 — SP-4 PR-A guard scrape series

`ainfera_dispatch_without_capture_total` registered; SP-4 PR-A's `DispatchCaptureCounter` plugs in via a 1-line `.inc()` once both PRs merge.

Privacy guardrails (SP-5 §1)

• NO tenant_id / agent_id / owner_handle on any metrics label
• /metrics is internal-key gated, not public
• Log lines scrubbed by KEY (api_key/password/prompt/messages/content/etc.) AND SHAPE (regex)

Tests

• `test_structured_log.py` — 10 cases (Ainfera key, OpenAI/Anthropic, Bearer, JWT, structured extra, nested dicts, innocent passthrough, exception tracebacks)
• `test_metrics_registry.py` — 13 cases (primitives, label escaping, cumulative buckets, sorted render, named-series wrappers)

Pre-commit: ruff + ruff-format + mypy --strict + pytest unit+smoke = 529 green.

Test plan

• CI green
• Branch / preview: `curl -H "X-Ainfera-Internal-Key: $KEY" /metrics` → 200 with the named series
• Without the header: `curl /metrics` → 401
• A few real requests then re-scrape — `ainfera_http_requests_total` increments for the seen routes; histogram populates bucket counts
• Log line in Railway shows JSON shape `{ts,level,logger,msg,...}` and no secrets appear

🤖 Generated with Claude Code

---

Note

Medium Risk
Touches global logging configuration (logging.basicConfig(..., force=True)) and adds always-on request instrumentation, which can affect log consumers and add minor runtime overhead; /metrics also introduces a new internal-authenticated endpoint and DB reads on scrape.

Overview
Adds an internal-key–gated GET /metrics endpoint that exposes a small dependency-free Prometheus registry, including per-request counters/latency histograms and a cold-path refresh of audit-chain gauges via max(seq)/max(created_at) DB reads.

Installs a global StructuredJSONFormatter in main.py (forced root handler) to emit JSON logs and scrub secrets/PII from both structured extra fields and freeform message/traceback text, and adds RequestMetricsMiddleware to record request counts/durations using route templates with a path-cardinality cap.

Includes new unit tests covering the metrics primitives/rendering and the log-scrubbing contract.

^{Reviewed by Cursor Bugbot for commit c153ecc. Bugbot is set up for automated code reviews on this repo. Configure here.}

[linear-code]

AIN-238 [Foundation] Cross-framework observability — one trace ID across all hops (atop L4 audit)

Pressure-test finding #12 + Ainfera OS design principle #6 (LOCKED 2026-05-22). Seven frameworks + the gateway + Letta + Modal + Spark-local models means a single request can cross 4-5 heterogeneous runtimes (e.g. Yavanna→Aulë→gateway→Modal→audit). When something breaks, tracing it across framework boundaries is an "observability black box" — the dominant 2026 multi-agent production failure. The L4 audit chain captures inference calls; it does NOT give end-to-end cross-agent traces.

What it must do

• One trace ID per logical request, propagated across every hop — agent → gateway → backend → audit — regardless of framework. OpenTelemetry-style context propagation, injected at the gateway and carried through handoffs (ties to AIN-233 handoff contracts).
• Per-hop spans — each agent, each framework call, each backend dispatch emits a span with timing + status, parented to the trace.
• Correlate with L4 audit — the audit chain's inference records link to the trace ID so a financial/audit event maps to the full request path.
• Failure localization — when a request fails or stalls, the trace shows exactly which hop (which agent, which framework, which backend) — no more guessing across runtimes.
• Feeds AIN-234 — per-agent cost/latency metering rides the same trace context.

Acceptance criteria

• Trace ID generated at gateway ingress, propagated through all agent + framework + backend hops
• Each hop emits a parented span (timing + status); visible in one trace view
• Trace ID correlated into L4 audit chain records
• Induced failure at any hop is localizable from the trace alone
• Works across all 7 frameworks (heterogeneous runtimes) + Modal + Spark-local

Why M1

Without cross-framework tracing, debugging a 7-framework fleet on Spark is guesswork. Pairs with AIN-234 (metering rides the same context) + AIN-233 (trace propagates via handoff contracts) + AIN-226 (gateway is the injection point).

AIN-249 [Routing] CONDITIONAL M_allowed wiring — brand_verdicts schema + RoutingRequest.context + explicit PASS rows

Code + schema follow-on to AIN-248. Wires the engine to consume CONDITIONAL M_allowed verdicts end-to-end. PASS/VETO/None already work today (Candidate.m_allowed 3-state); this ticket adds the context-aware path.

⛔ Item #1 — SCHEMA LOCK (Discipline #12, founder ruling required before any code)

Same shape of one-shot immutable decision as F4. Recommended: separate brand_verdicts table (not brands.verdict cols).
Rationale: (1) a verdict is an audit record (reviewed_by/approved_by/approved_at), not a brand attribute; (2) CONDITIONAL is context-dependent → a brand may need multiple/superseded verdict rows (per-jurisdiction, over time) — columns can't hold history cleanly; (3) keeps catalog clean, matches append-don't-mutate ethos. Brain reads a current-verdict view; table keeps the trail.
→ Do not start items 1–3 until founder rules table-vs-columns.

Scope (≈150 LOC once schema lock taken)

1. Verdict schema (Alembic). brand_verdicts: {brand_id, verdict: pass|conditional|veto, condition: jsonb, hosting_endpoint: text, dimensions: jsonb, counsel_required: bool, reviewed_by, approved_by, approved_at}. RLS posture matching the catalog family. Engine reads only resolved verdict + condition; dimensions/counsel_required/reviewed_by/approved_by are audit metadata the brain does not read.
2. Request-context model. RoutingRequest (ainfera_routing.types) gains context: dict (e.g. tenant_jurisdiction, hosted_region). Enrolment gate evaluates the brand's condition against context → resolves m_allowed. Additive — callers that pass no context default to the strictest interpretation (CONDITIONAL without satisfying context → deny).
3. Explicit PASS rows for the 5 active brands. Replace today's implicit "pass via brand.active=true" (api/ainfera_api/services/routing_brain.py:174) with first-class verdict rows so the audit trail shows why allowed, not just that it is. approved_by: founder from launch.

NOT in scope

• Ulmo intake + founder-signed verdicts themselves = AIN-248 (human/counsel-facing, zero code).
• D7 SBOM gate (AIN-237), EU Annex IV bundle = operational gates before a verdict lands, not engine inputs.

Done bar

• Schema lock ruled + migration applied (Alembic, dev-branch tested)
• context field live; CONDITIONAL resolves correctly for a test brand (satisfied context → candidate; unsatisfied → M_ALLOWED_VETO)
• 5 active brands have explicit PASS verdict rows; routing unchanged for them (regression-clean vs current)
• routing_outcomes.m_allowed_set captures the resolved boolean + condition outcome
• No-context request still routes (strictest-default proven)

Blocked by: AIN-248 (needs first verdicts to test against). Related: AIN-245 (engine), AIN-237 (D7).

Review in Linear

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is ON. A cloud agent has been kicked off to fix the reported issue.}

^{Reviewed by Cursor Bugbot for commit 284d6eb. Configure here.}

[cursor]

Metrics opens DB before auth

Medium Severity

GET /metrics declares DBSession as a handler dependency while the internal-key check runs only inside the function body. FastAPI resolves dependencies before the handler runs, so missing or invalid X-Ainfera-Internal-Key requests still acquire a database session even though no query should run.

 

^{Reviewed by Cursor Bugbot for commit 284d6eb. Configure here.}