feat(ai-observability): OTLP traces, waterfall, and Valkey correlation by KIvanow · Pull Request #313 · BetterDB-inc/monitor

KIvanow · 2026-07-13T06:14:29Z

Summary

Adds OTLP trace ingestion and an "AI Traces" waterfall that shows the BetterDB cache and memory spans inside each request, then correlates them with live Valkey state to explain why a hit or miss happened. Something a pure trace tool cannot do.

Stacked on #312 (review that first).

Changes

otel_spans storage plus per-trace summary aggregation across sqlite / postgres / memory, with retention.
OTLP POST /v1/traces ingestion (JSON), keeping @betterdb/* spans plus roots. Optional bearer token and enable gate.
Trace query API: GET /ai/traces and GET /ai/traces/:traceId.
TraceCorrelationService and GET /ai/traces/:traceId/correlate: joins spans to live EXISTS/TTL, instance threshold, and index state.
Web: "AI Traces" page (trace list, waterfall, span panel with the correlation).

Checklist

Unit / integration tests added
Docs added / updated
Roborev review passed. Run roborev review --branch or /roborev-review-branch in Claude Code (internal)
Competitive analysis done / discussed (internal)
Blog post about it discussed (internal)

Note

Medium Risk
Cloud OTLP ingest bypasses session auth and relies on bearer tokens plus startup validation; misconfiguration could allow anonymous writes. New storage/query paths and correlation issue live Valkey commands on user-selected traces.

Overview
Adds OTLP/HTTP JSON trace ingestion at POST /v1/traces (outside the api prefix), with optional bearer auth, an enable gate, and fail-closed behavior in cloud when OTEL_INGEST_TOKEN is unset. Ingest keeps @betterdb/* spans plus trace roots and drops other instrumentation.

Persists spans in a new otel_spans layer on memory/sqlite/postgres (save, per-trace summaries, span lists, whole-trace pruning) and wires cloud + self-hosted retention for that data.

Exposes GET /ai/traces, GET /ai/traces/:traceId, and GET /ai/traces/:traceId/correlate (live Valkey EXISTS/TTL, thresholds, index state with human explanations). The web app adds an AI Traces page (list, waterfall, span detail with correlation).

Cloud auth now allowlists /v1/traces and treats CLOUD_MODE === 'true' consistently; startup env validation requires OTEL_INGEST_TOKEN in cloud mode.

^{Reviewed by Cursor Bugbot for commit bca14aa. Bugbot is set up for automated code reviews on this repo. Configure here.}

jamby77 · 2026-07-14T10:37:28Z

These new vars are read straight from process.env — OTEL_INGEST_ENABLED, OTEL_INGEST_TOKEN here and AI_OBS_POLL_INTERVAL_MS in #312. Convention is Zod-validated env in config/env.schema.ts + an entry in .env.example. OTEL_INGEST_TOKEN matters most — it's the only credential guarding cloud ingest, shouldn't be silently optional.

jamby77

LGTM. Env-var note above is non-blocking.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d96ebb4. Configure here.}

KIvanow · 2026-07-14T11:43:56Z

Addressed.

OTLP span storage foundation for the trace waterfall. - @betterdb/shared: StoredOtelSpan / OtelTraceSummary / OtelTraceQueryOptions. - otel_spans table + StoragePort methods (saveOtelSpans, getOtelTraces, getOtelTraceSpans, pruneOldOtelSpans) across sqlite / postgres / memory, deduped on (trace_id, span_id). getOtelTraces aggregates per trace: root span name/service/duration (fallback to max-end minus min-start), span count, @betterdb/* span count, and an error flag. Times kept as unix-nano strings. - Retention wired into the tier-based data-retention sweep. 10-test cross-adapter spec (Memory + SQLite). Typecheck clean.

- OtelIngestService: parses OTLP/JSON ExportTraceServiceRequest, keeps spans from @betterdb/* scopes plus root spans (for parent context), flattens attributes, converts nano timestamps, and persists via saveOtelSpans. - OtelIngestController: POST /v1/traces (excluded from the global `api` prefix so it's the OTLP-standard path). Optional bearer auth (OTEL_INGEST_TOKEN), gate (OTEL_INGEST_ENABLED). JSON encoding first; protobuf is a follow-up. - Query API on AiObservabilityController: GET /ai/traces (recent summaries), GET /ai/traces/:traceId (span tree for the waterfall). 3 ingest tests (16 in the module). Typecheck clean.

Adds the "AI Traces" page: recent-traces list (root name, service, duration, span/BetterDB counts, error flag) → per-trace waterfall (depth-indented span rows with timeline bars, BetterDB spans highlighted) → span attribute panel. Cache-miss spans surface a correlation hint linking to the AI Cache & Memory tab. Route /ai-traces + sidebar nav; api/aiObservability.ts gains getTraces / getTraceSpans. Web typecheck clean. Full server-side span↔Valkey-state correlation and OTLP protobuf are follow-ups.

… state The differentiator: TraceCorrelationService joins a trace's @betterdb/* spans with the live Valkey instance. It parses cache.key / cache.matched_key, runs EXISTS/TTL, reads the instance threshold (__config / __mem_config) and FT index state, and emits an explanation (cold-miss-now-warm, still-uncached, hit-since-expired, index-degraded). New GET /ai/traces/:traceId/correlate and a SpanCorrelation shared type; the AiTraces span panel renders it. 4 correlation tests (20 in the module). Typecheck clean (api + web). Caveat: assumes the connected Valkey is the same instance the app used.

Wires the real MemoryAdapter + OtelIngestService + AiObservabilityService + TraceCorrelationService and drives a realistic chat.turn OTLP/JSON payload through ingest → store → query → correlate. Asserts scope filtering (ai.streamText dropped, root + 3 @betterdb spans kept), trace-summary aggregation, span-tree ordering, and per-span correlation (cold-miss-now-warm, threshold/index read).

Addresses Bugbot findings on the OTLP traces PR. - getOtelTraces now filters whole traces by their trace-level start / service (subquery on grouped traces), then aggregates ALL spans of the matching traces, across sqlite / postgres / memory. Previously spans outside the window were dropped before aggregation, so a boundary trace showed a partial summary (wrong spanCount, missing root, understated duration). Adds a regression test. - Postgres saveOtelSpans now upserts (ON CONFLICT DO UPDATE) to match the sqlite / memory last-wins semantics, so a re-ingested span updates.

…es in cloud auth Bugbot follow-up on the OTLP traces PR. - otel-ingest: many exporters send an all-zero parentSpanId for root spans; treat empty OR all-zero as root so a non-@betterdb root (e.g. chat.turn) is not dropped and traces keep their root/waterfall. Adds a test. - cloud-auth: allowlist /v1/traces in the middleware and guard so machine OTLP exporters aren't redirected by the browser-session gate in CLOUD_MODE. The endpoint has its own OTEL_INGEST_TOKEN bearer auth (like /agent/ws, /mcp/).

…orrelation key Bugbot follow-up on the OTLP traces PR. - maybePruneLocally now also trims otel_spans (7d) on self-hosted, so ingested spans don't accumulate forever off-cloud. - pruneOldOtelSpans now deletes WHOLE traces (by trace-level min start) across sqlite/postgres/memory, so retention never orphans a trace's later spans while dropping its root. Adds a regression test. - AiTraces correlation refetchKey now includes the connection id, so switching Valkey connections doesn't show the previous connection's correlation.

Bugbot follow-up. A single multi-row INSERT ... ON CONFLICT DO UPDATE errors on Postgres ("cannot affect row a second time") when the same (trace_id, span_id) appears twice in one batch, dropping the whole export. saveOtelSpans now dedupes the batch (last wins) before building the insert. Adds a same-batch-duplicate test (Memory/SQLite already tolerate it; this guards the Postgres path in CI).

Bugbot follow-up. TraceCorrelationService only correlated spans with a cache.key/matched_key, so semantic-cache (and memory) MISS spans — which report cache.name + cache.hit=false but no key — were skipped, exactly the case the feature targets. Now derive the instance from cache.name when there's no key, skip the EXISTS/TTL lookup (keyExistsNow/keyTtlSeconds stay null), and still surface the instance threshold + index state with a miss-appropriate explanation. Adds a keyless-miss test.

Bugbot follow-up. activeTrace kept using selectedTrace even when that id was no longer in the (1h-windowed) traces list, so the waterfall could show a trace absent from the sidebar and keep polling it. It now honors the selection only while it's still in the list, else falls back to the newest trace.

Ingest keeps only @betterdb spans and trace roots, so many BetterDB spans reference a parent that was never stored. computeDepths treated a missing parent as depth 0, lining those spans up with the root instead of as its descendants. Distinguish true roots (no parentSpanId) from orphans (parent set but absent) and nest the latter one level in.

In CLOUD_MODE /v1/traces is allowlisted past session auth, so the bearer token is the only credential. The controller only enforced it when OTEL_INGEST_TOKEN was set, leaving the endpoint open to anonymous span writes into a tenant's store when the var was unset. Require the token in cloud mode and reject with 401 when it is missing.

… in cloud Add OTEL_INGEST_ENABLED and OTEL_INGEST_TOKEN to the Zod env schema and .env.example. Enforce OTEL_INGEST_TOKEN at boot when CLOUD_MODE is set: /v1/traces is allowlisted past session auth there, so the token is the only credential guarding tenant ingest and must not be silently optional. The controller keeps its runtime 401 guard as defense in depth.

…bounds CLOUD_MODE truthy mismatch: the OTLP ingest token guard (controller + env schema) treated any non-empty CLOUD_MODE as cloud, while retention and the rest of the codebase use CLOUD_MODE === 'true'. A value like 'false'/'0' would demand the ingest token yet skip cloud retention. Normalize the whole /v1/traces auth path to === 'true' (controller, env schema, and the cloud-auth middleware/guard that allowlist the path) so the session-auth bypass and the token requirement stay in lockstep. Also clamp GET /ai/traces hours to 168 and limit to 1000, matching the history endpoint, so a large window can't force full-table aggregation.