feat(observability): LangGraph span panel with Postgres persistence#6
Merged
Conversation
oxlint --type-aware widening of status?.type breaks the toolStatusIconMap index signature. The status fallback through ?? was inferred as any, so the literal type ToolStatus (= ToolCallMessagePartStatus['type']) needed an explicit annotation.
Pulls the new Timeline + TimelineBar primitives (which our panel ends up not using, since we went the custom SVG path to match the official waterfall example) and any compatible api-surface tweaks between 0.0.24 and 0.0.25.
…-derived parents CapturingHandler in backend/observability derives the parent chain from `langgraph_checkpoint_ns` (strip trailing `|name:uuid`) instead of trusting LangChain's `parent_run_id`, which lies under USE_SUBGRAPH. Frontend `captured-to-span-data` is now read-only — it only walks `parent_span_id`. The preview page renders two Panels (subgraph / inlined) with independent Sheet state so we can compare both topologies against the same prompt. TraceView/TraceCard removed from observability-panel; only waterfall remains.
… LLM input to {prompts}
- time_to_first_token_ms: LLM-only metric moves from top-level into meta so
§9.1 columns stay stable. LC itself doesn't emit it; handleLLMNewToken
writes it.
- input: handleChatModelStart / handleLLMStart now write
{prompts: string[]} instead of bare string[]. Chat-model BaseMessage[][]
is stringified by stringifyMessages first. Tool/chain/retriever inputs
untouched.
- chat_model: removed. meta.ls_model_type is the LC-native equivalent
(chat/llm/embedding); serialized_llm already carries the class id. Field
was always true in our graph, pure noise.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
unwrapLCMessage previously fell back to className.toLowerCase() when ROLE_BY_CLASS had no entry, which leaked garbage like role: "e", role: "0", role: "d" into captured messages. The trigger: LC V2 streaming chunks emit envelopes whose id array is a single element ([<chatcmpl-uuid>] or [<message-uuid>]), not the standard [langchain_core, messages, HumanMessage]. Those id arrays are not class names — they're run ids. PascalCase regex on the last id was already filtering most of them out, but isLCMessageEnvelope still returns true for V1 envelopes with single-element id arrays (kwargs.id becomes the array), so unwrapLCMessage fires and produces nonsense. Fix: when className isn't in ROLE_BY_CLASS, omit role entirely. The message keeps its content / tool_calls / id; frontend renders the content without a role chip. Verified: /tmp/captured-spans-fix.json no longer contains role fields with single-character values; HumanMessage / AIMessageChunk / ToolMessage (V1, three-element id) still unwrap to user/ai/tool. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…/ flat paths
Earlier implementation had two issues:
1. ROLE_BY_CLASS derived role from id / lc_namespace last element —
these arrays are sometimes chatcmpl UUIDs or message UUIDs, not
class-name paths, producing role:"e" / role:"0" / role:"d"
garbage in captured messages.
2. The unwrap returned {role} only when ROLE_BY_CLASS matched; for
V2 chunk envelopes with single-element id arrays, we silently
dropped role (and after a later fix, accidentally dropped ALL
fields because the unwrap path wasn't matched).
Replace with four explicit paths:
- live BaseMessage instance: Symbol.for("langchain.message") = true.
Dominant path in our graph — LC doesn't .toJSON() messages that
survive through the reducer, so handleChainEnd sees live instances.
Pull content / tool_calls / id / additional_kwargs / response_metadata
directly from instance own properties.
- V1 envelope: {lc:1, type:"constructor", id:[...], kwargs:{...}}.
Serialized form via Serializable.toJSON().
- V2 envelope: {lc_serializable:true, lc_namespace:[...], lc_kwargs:{...}}.
Naive JSON of a live instance crossing a layer that didn't call
.toJSON() (streaming chunks).
- flat: plain object — caller passes through unwrapMessage returning
null, deepUnwrapLC keeps it as-is.
Role authority: kwargs.type (V1) > lc_kwargs.type (V2) > top-level
type (live instance) > top-level role (flat). Class name derived
from id / lc_namespace is NEVER used.
All 12 captured messages now have content / tool_calls / id where
expected (vs 1/12 before).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Field exposed baseURL (internal proxy) and field names hinted at credential locations; the redact regex only matched *_api_key / *_secret / *_password, so URLs and org IDs leaked through. Model identity is already covered by ls_model_name + ls_model_type + ls_provider + serialized_llm — the kwargs blob is duplicated state with no observability value.
Spec-kit artifacts for the next phase of the observability panel MVP (after the callback-collector landed in commit 211d133). Scope: persist captured spans in Postgres, expose them via a thread-scoped API, wire the handler into the graph, and add a UI sheet. Retention is configurable via OBSERVABILITY_RETENTION_DAYS (default 30) and surfaced in the panel as a banner. Key design decisions captured: - DB writes follow renameThreadAgent's pattern: buffer in callback handler, await bulkInsert on handleChainEnd (avoids per-token round-trips). - Route is nested under the thread resource at /api/threads/[id]/observability with withAuth + getThreadForUser ownership check; cross-user access returns 404 to avoid enumeration. - Sensitive field filter (FR-009) rejects api_key / baseURL / Bearer before insert; defensive layer on top of the llm_kwargs removal in 211d133. - Retention physical delete script is deferred to the polish phase; banner + env + GET response field land in Phase 2.
CapturingHandler attaches at compile().withConfig() so ToolNode receives callbacks; the previous model.withConfig path only covered the LLM node. Sheet is rendered once at ThreadRoot and opened via context ref; per-message buttons no longer mount their own Sheet instance. transform.ts adds parentIdFor + clampCycles to stop SpanResource.calculateDepth from recursing on long LLM chains. Phase 1 payload trims drop duplicate messages, redundant meta keys, and trim tool output to head+tail. Rename scripts/retention.ts to scripts/cleanup-observability.ts. Delete dev-only preview paths per FR-012. refs specs/002-observability-panel/spec.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Promote parent_message_id from meta jsonb to a first-class nullable column with composite btree index so the per-turn API can filter via path param (GET /api/threads/:id/observability/:parentMessageId). The column becomes the single source of truth: bulkInsertSpans backfills null rows from the column at INSERT time, findLatestParentMessageId reads the column directly, and CapturingHandler.start() writes the in-memory field but no longer races an async DB lookup — sync Start hooks stay sync. Frontend drops the parentMessageId thread-through; backend derives the id from inputs.messages (interrupt resume / regenerate / cold-start rows get null, the pre-INSERT backfill catches them). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire end-to-end interrupt observability: handleToolError stamps meta.interrupt_tool on the synthetic human span, bulkInsertSpans' pre-INSERT backfill flips any prior waiting human span on the same thread+tool to completed (replaces the dropped in-memory openHumanSpanId that died on langgraphjs dev restart). Transform layer accepts kind=human leaves and maps status=waiting to running for the panel. Panel drops the per-row status dot, the (step N) suffix, and adds a blue legend chip for human. Also: replace db:migrate with scripts/migrate.ts driving drizzle-orm/postgres-js/migrator directly (drizzle-kit spinner exit 1), backdate journal when so the migrator's created_at comparison skips the already-applied 0001 row. Fix pre-existing @next/env named-import in cleanup-observability.ts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Share a TypeChip between the row badge and the legend so the type swatch in the bottom strip matches the per-row chip. Each kind gets a lucide icon in the per-type color: BrainIcon (llm), WrenchIcon (tool), BoxIcon (node), LinkIcon (chain), UserIcon (human). Legend order is pinned via LEGEND_TYPE_ORDER = [chain, node, llm, tool, human]. WaterfallBar now reads useSelection() and strokes the bar of the selected row so the highlight on the left label tracks the bar on the right. Chip and name use leading-none + a tight py so the chip sits on the name's baseline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the name-search header with an 8-tile stat card grid (duration, token in/out/cache, TTFT avg/max, LLM/Tool/Failed counts) sourced from aggregateRoot(). Move retention footer from the sheet header to the bottom of the waterfall, English copy. Surface meta.ls_model_name on LLM rows so multi-provider threads read model-first instead of 'ChatOpenAI' / 'ChatAnthropic'.
…nsive sheet Side-by-side on lg+ (waterfall left, details right when selected); mobile keeps top-to-bottom. Sheet width is w-full on mobile and md:w-3/4 on tablets+ — uses !max-w-none to escape the right-variant's sm:max-w-sm clamp. formatDuration always renders as seconds (xx.xxs). Add Waiting stat card (ClockIcon + human blue, only when status=waiting span exists) and replace Cache read with Total (input + output, excluding cache_read to avoid double-counting).
trimMeta initializes time_to_first_token_ms to null at start() time. The handleLLMNewToken guard was checking only === undefined, so the first token never set the value and every LLM row persisted as null. Accept both null and undefined so the first token writes the timestamp.
Replace per-kind if/else renderers with a single FieldValue + FieldDef schema (DETAIL_SECTIONS_BY_KIND) walked by resolveSections(). The same schema feeds the hover tooltip (tooltip:true opt-in, details:false opt-out) so both surfaces share one source of truth.
- Header: kind chip via TypeChip, two-row layout, StatusBadge right-aligned, @{node} no space, no duration dup
- Status badge shrunk (text-[9px] py-0) to match kind chip weight
- LLM token breakdown: 3-row layout, no progress bars, inline cache/reasoning; total row borderless
- Model badge in LLM color
- bare:true row flag for single-field sections
- details:false row flag (CONTEXT Tokens now tooltip-only, lives in Total cost breakdown)
- Node/chain/human Name row hidden when name == node
- Tooltip auto-flips to left when overflows right edge (useLayoutEffect + offsetWidth)
- Tooltip KV: values right-aligned (truncate text-right, badge via flex justify-end)
- Inline CopyJsonButton on JsonBlock + Fields (Fields copy moved to right of K column, not absolute over rows)
- Start/End show YYYY-MM-DD HH:MM:SS.mmm so cross-midnight traces are distinguishable
Drop the failedCount > 0 / waitingCount > 0 guards on the two status-summary StatCards so a clean run still surfaces a 0-valued placeholder. Rename Waiting to Human in the loop to match what the underlying kind=human / status=waiting span actually is (LangGraph interrupt marker), not the generic 'waiting' verb. Also drop the dead aggregateUsage helper and the unused ZapIcon import — both flagged by oxlint type-aware as never-read. Update the interrupt test to assert ended_at is non-null: the collector stamps Date.now() so markRunningAsFailed (which keys on ended_at === null) does not mis-attribute the interrupted tool as an aborted run. The synthetic human span alongside the tool carries the wait semantics; the tool's ended_at just needs to be present.
- OBSERVABILITY.md: update security (redact not throw), callback wiring (agent.ts compile layer not model.ts), threadIdOf dual-compat field, parent_message_id/human/waiting schema additions, per-turn sub-route, correct script path (cleanup-observability not retention), add curl examples (SC-007) - APIS.md: update observability section header (agent.ts wiring) - CLAUDE.md: remove stale assistant-ui/observability-*.tsx paths, add per-turn sub-route entry - README.md: add observability feature bullet, project layout entries, OBSERVABILITY_RETENTION_DAYS env var, docs link - TODOS.md: remove completed observability entry (resolved entries deleted per file policy)
scripts/migrate.ts was a workaround for drizzle-kit hanging on an ora spinner. Removing it and using drizzle-kit migrate directly.
FireTable
changed the title
This was referenced Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the full observability panel feature (spec002): every LangGraph
invoke now captures LLM / Tool / Chain / Node / Human spans via a
BaseCallbackHandler, persists them to aobservability_spansPostgrestable, and surfaces them in a per-turn waterfall Sheet accessible from
each assistant message.
What's included
Backend
backend/observability/callback-collector.ts—CapturingHandler:buffers in-flight spans per
runId, unwraps 4 LC message formats,derives
parent_span_idfromlanggraph_checkpoint_ns(LC'sparent_run_idis unreliable under subgraphs), captures TTFTbackend/agent.tsvia.withConfig({ callbacks: [capturingHandler] })— ensures ToolNodespans are captured (model-level wiring misses them)
human/waitingspan kind+status for GraphInterrupt gaps;backfillWaitingInterruptSpanscloses gaps DB-side on resumeAPI
GET /api/threads/[id]/observability— all turns merged,withAuth+ownership check → 404, preflight
markRunningAsFailedGET /api/threads/[id]/observability/[parentMessageId]— per-turnfilter using the
parent_message_idbtree indexDELETE /api/threads/[id]/observability— clear thread spansDB
observability_spanstable withparent_message_idcolumn (denormalizedfrom
metafor indexed per-turn filtering),humankind,waitingstatus;
ON DELETE CASCADEfromthreads(id)(thread_id, started_at),(thread_id, parent_message_id, started_at),(created_at)for retentionFrontend
<ObservabilitySheet>atThreadRoot(no per-message N-foldfetch);
<ObservabilityButton>passesparentMessageIdfor per-turnview
lib/observability/transform.ts—CapturedSpan → SpanDatafor@assistant-ui/react-o11ywaterfall rendererkind chips, detail expansion, retention footer
Security
bulkInsertSpansredacts forbidden fields (api_key,baseURL,organization, Bearer tokens) — first 5 chars preserved, rest***;console.warnfor auditability (redact-not-throw to avoid false positiveson user prose)
Retention
OBSERVABILITY_RETENTION_DAYSenv var (default 30, must be positive int)scripts/cleanup-observability.ts— cron-ready physical delete scriptDocs & cleanup
docs/OBSERVABILITY.md— full design doc (security, schema, retention,curl examples)
docs/APIS.md,README.md,CLAUDE.md,docs/TODOS.mdupdatedscripts/migrate.ts(replaced bydrizzle-kit migrate),components/assistant-ui/mock-spans.ts(unused)Tests
tests/lib/observability/queries.test.ts— insert / get / markFailed /delete / redaction (28 tests)
tests/api/threads/observability.test.ts— GET 200/401/404 / DELETE(13 tests)
tests/api/threads/observability-parent.test.ts— per-turn route(6 tests)
tests/backend/observability/callback-collector.test.ts— 4 LC messageformats / parent derivation / TTFT / bulkInsert wiring (16 tests)
tests/integration/observability-wiring.test.ts— end-to-end wiring(2 tests)
References
specs/002-observability-panel/spec.mddocs/OBSERVABILITY.md