feat(telemetry): add traces and logs to the telemetry package#901
Merged
Conversation
Add tracer provider and logger provider sharing the resource and OTLP endpoint/headers config with the existing meter provider. Wire tool call spans in the tool service decorator with GenAI semconv attributes. Set the global W3C trace-context propagator so SDK client requests carry the traceparent header. Add StartLLMTurnSpan and StartSessionSpan helpers for agent code to use. All three signals honor OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME, and OTEL_RESOURCE_ATTRIBUTES; zero overhead when telemetry.enabled=false.
This was referenced Jul 13, 2026
edenreich
marked this pull request as ready for review
July 13, 2026 15:34
The span helpers shipped unplugged: no session root span, no LLM-turn
spans, and every tool span was an orphan single-span trace. Wire them:
- Recorder.StartSession begins the root span (cmd/agent.go and the chat
TUI session in cmd/chat.go) and returns an end func that stamps
infer.run.outcome; SpanContext grafts the root span onto request
contexts, since the chat TUI builds per-message contexts that don't
descend from session start.
- LLM-turn spans wrap the sync call in AgentServiceImpl.Run and each
streaming request in streamOnce; headless tool calls now derive from
the session context, so execute_tool spans nest in the same trace.
- Nil-recorder paths return noop.Span{} instead of the span already in
ctx, so a stray defer span.End() can never end a caller's span.
- Failed tool spans set codes.Error status alongside error.type.
- Build the resource once and share it across meter and tracer providers;
drop the always-nil error returns from newTracerProvider.
- Aggregate skips -traces.jsonl files (they share the dir so Archive
retention covers them, but hold no metrics).
Descoped to keep #893 honest, remain open there: the logs signal (needs
a zap-to-OTel bridge; the provider had no producer and wrote empty files
every session) and W3C traceparent propagation (the SDK has no OTel
awareness, so setting a global propagator put nothing on the wire).
This was referenced Jul 13, 2026
…ntions
Audited against the GenAI span semantic conventions
(open-telemetry/semantic-conventions-genai), the trace concepts doc, and
the OTLP exporter env-var spec:
- execute_tool spans set gen_ai.operation.name=execute_tool (Required).
- Inference spans use CLIENT kind (remote gateway call), stamp
gen_ai.usage.input_tokens/output_tokens (via storeIterationMetrics, the
chokepoint both sync and streaming paths share), and set error.type +
Error status on failed turns per the recording-errors rules.
- Session and inference spans carry gen_ai.conversation.id (session id).
- Resource merges resource.Default() so the required telemetry.sdk.*
attributes are present.
- OTLP exporters honor the spec's env-var precedence: a config endpoint
still wins, but when absent the exporter's native handling applies
(per-signal OTEL_EXPORTER_OTLP_{TRACES,METRICS}_ENDPOINT over the
generic var, correct per-signal path appending, headers, timeouts). A
path-less config endpoint is passed as a base so each signal appends
its own /v1/<signal> path.
This was referenced Jul 13, 2026
edenreich
added a commit
that referenced
this pull request
Jul 13, 2026
## Summary Closes #903. PR #901 (towards #893) records the traces signal locally — every session writes a per-session `~/.infer/telemetry/<session>-traces.jsonl` with a `session` root span, `chat <model>` spans per LLM turn, and `execute_tool <name>` spans per tool call — but nothing in the CLI rendered them. This adds a local span-tree viewer, paired the same way `infer stats` / `/stats` are. ## What's included - **`infer traces [session-id]`** renders the span tree of a session (root → LLM turns → tool calls) with per-span durations from the local trace file; with no argument it picks the most recent session. `--list` enumerates sessions that have trace files, `--format json` emits the tree as structured output (span names, start times, fractional-ms durations, attributes, children). - **`/traces [session-id]`** chat shortcut renders the same tree inside the chat TUI, colored from the static CLI palette (dim connectors, accent durations, red error markers). The output is deliberately plain text rather than markdown: the chat markdown renderer intentionally bypasses glamour for content containing box-drawing characters, so fenced/heading markup would display raw. - **Error visibility**: spans with error status or an `error.type` attribute carry a trailing `[error: <type>]` marker, rendered red in the CLI. - **Live sessions**: spans export as they end, so a running session's file has no `session` root yet. Orphan spans sharing a missing parent are grouped under a synthetic `session (in progress)` root, so mid-session `/traces` still renders a proper tree instead of a flat list. - **Duration fidelity**: durations are kept at microsecond precision (`162µs`, `41ms`, `3.4s`, `1m32s`) instead of truncating sub-millisecond tool spans to `0ms`. ```text session (standard, success) 38.8s ├── chat ollama_cloud/deepseek-v4-flash 3.4s ├── execute_tool Read 162µs ├── execute_tool Read 137µs ╰── chat ollama_cloud/deepseek-v4-flash 8.1s ``` ## Implementation notes - The tree is rendered via `charm.land/lipgloss/v2/tree` with a `TreeStyle` of `lipgloss.Style` values; the zero value renders plain, which is what the shortcut's non-TTY-safe paths rely on. Widths are measured with `lipgloss.Width` and durations are right-aligned across the whole tree. - `LoadTraceTree` links spans purely by `Parent.SpanID`, order-independent, so once subprocess span ingestion lands (#904) child-process spans appear under their tool span with no viewer changes. - Read-only over the existing stdouttrace stub format already exercised by `internal/telemetry/tracer_test.go`; works fully offline, no OTLP endpoint required, and displays no prompt/response content beyond existing span names/attributes. ## Testing - Table-driven unit tests for tree assembly (sorting, orphan grouping, malformed lines, error-status precedence), session listing, and duration formatting, plus an end-to-end test that records a real session → LLM turn → failing tool trace through the `Recorder` and asserts the loaded tree, keeping the viewer in lock-step with the writer's format. - Shortcut tests for empty store, default-to-latest, explicit session id, and unknown session. - Verified live in the chat TUI (tmux + mock gateway) and against real recorded sessions via the built binary. ## Follow-ups - `[DOCS]` issue against inference-gateway/docs for the new command/shortcut (CLI reference + shortcuts page). - #904 ingestion must persist received OTLP spans into the parent session's file in the stdouttrace stub format with hex span ids for parent linking to work.
Contributor
|
🎉 This PR is included in version 0.147.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Towards #893
Summary
Adds the traces signal to the telemetry package, wired end to end: one root
sessionspan per agent run, achat <model>child span per LLM request (sync and streaming), and anexecute_tool <name>child span per tool call - all in a single trace, with GenAI semantic convention attributes and no prompt/response content. The tracer provider shares the resource and OTLP endpoint/headers config with the existing meter provider.Verified against the mock gateway (
INFER_GATEWAY_MOCK=true infer agent): one trace id per session,sessionroot carryinginfer.execution.mode/infer.agent.mode/infer.run.outcome, turn and tool spans nested beneath it,error.type+ Error status on a failed tool call.Changes
<session>-traces.jsonl, synchronous on purpose - spans survive an abrupt CLI exit) and opt-in OTLP/HTTP export.StartSession(mode)begins the root span and returns an end func that stampsinfer.run.outcome;SpanContext(ctx)grafts the root span onto request contexts (the chat TUI builds per-message contexts that don't descend from session start);StartLLMTurnSpanfor per-request child spans;SetSpanUsage/SetSpanErrorhelpers. All nil-safe: a nil Recorder returnsnoop.Span{}, never the caller's span. Resource built once and shared by both providers.gen_ai.operation.name=execute_tool,gen_ai.tool.name,gen_ai.tool.type,infer.tool.outcome, pluserror.typeand Error status on failure.Aggregateskips-traces.jsonlfiles (they share the directory soArchiveretention covers them, but hold no metrics;infer statsoutput is unchanged).Runcall and around each streaming request instreamOnce; token usage stamped instoreIterationMetrics(the chokepoint both paths share);RunWithStreamgrafts the session span onto its session context.Aggregateignoring trace files.Standards conformance
Spans follow the OTel GenAI semantic conventions (audited against open-telemetry/semantic-conventions-genai): inference spans are CLIENT kind with
gen_ai.operation.name/gen_ai.provider.name/gen_ai.request.model,gen_ai.usage.input_tokens/output_tokens, andgen_ai.conversation.id; tool spans are INTERNAL withgen_ai.operation.name=execute_tool+gen_ai.tool.*; failed spans seterror.typeand Error status per the recording-errors rules. The resource includes the requiredtelemetry.sdk.*attributes.OTLP config follows the exporter env-var spec:
telemetry.otlp.endpoint(a base URL; each signal appends its own/v1/<signal>path) takes precedence, and when unset the exporters' native env handling applies - per-signalOTEL_EXPORTER_OTLP_TRACES_ENDPOINT/_METRICS_ENDPOINTover the genericOTEL_EXPORTER_OTLP_ENDPOINT, plus headers and timeouts.OTEL_SERVICE_NAMEandOTEL_RESOURCE_ATTRIBUTESmerge into the shared resource. Zero overhead whentelemetry.enabled=false.Descoped (remain open in #893)
LoggerProviderbridge; an earlier revision shipped a provider with no producer, which only created empty-logs.jsonlfiles every session.inference-gateway/sdkfirst. Subprocess/context propagation is tracked in [FEATURE] W3C trace context and baggage propagation to subprocesses, span ingestion via OTLP #904; the local span-tree viewer in [FEATURE] Add /traces shortcut and infer traces command (local span-tree viewer) #903.Docs follow-up: inference-gateway/docs#389