Skip to content

feat(dev): collect local OTEL traces in project dev - #1980

Open
tejaskash wants to merge 5 commits into
refactorfrom
feat/dev-otel-collector
Open

feat(dev): collect local OTEL traces in project dev#1980
tejaskash wants to merge 5 commits into
refactorfrom
feat/dev-otel-collector

Conversation

@tejaskash

@tejaskash tejaskash commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

What this does

When you run agentcore project dev, your agent now gets local tracing out of the box. The CLI starts a small local trace receiver, tells the agent's OpenTelemetry SDK to send traces to it (via environment variables), and saves everything to files under agentcore/.cli/traces/. So while you develop, every invocation of your agent leaves a trace on disk you can inspect — model calls, tool calls, timings.

Nothing is sent anywhere: the receiver only listens on 127.0.0.1 and writes to your project folder. If you don't want it, pass --no-traces or set instrumentation.enableOtel: false on the runtime.

Why we need it

This is the plumbing for the Agent Inspector (the local web UI from the current CLI). The Inspector's trace view reads exactly these files — without this PR there is nothing for it to show. It also stands alone: even before the Inspector lands, you get trace files you can cat/jq while debugging an agent.

Where the code comes from

This is a port of what the current agentcore-cli already ships in src/cli/operations/dev/otel/ — same mechanism, same on-disk format, same env vars. Differences from the original are structural, to fit the rewrite's layering:

  • The old OtelCollector class (HTTP + persistence + queries in one) is split into an HTTP receiver, a TraceStore, and pure transform functions — each independently testable.
  • The old CLI scans for a free port starting at 4318; here the OS assigns one, since the agent learns the address from an env var anyway.
  • The Python quirk is ported as-is: uvicorn's --reload respawns workers, so instrumentation is injected via sitecustomize.py on PYTHONPATH rather than the opentelemetry-instrument wrapper (which would only instrument the parent process, and traces would silently stop after the first hot reload).
  • Containers can't reach the host's 127.0.0.1, so their endpoint is rewritten to host.docker.internal — same as the original.

One dependency is added, pinned exactly: @opentelemetry/otlp-transformer@0.213.0. Agents send traces in protobuf (the Python SDK supports nothing else over HTTP), and this is the last version that ships the protobuf request decoders — the same version the current CLI uses. Don't let dependabot bump it.

Stack

Targets #1966 (feat/project-dev-handler); retarget to refactor after it merges. PR B (the Agent Inspector web UI) builds on this.

Verification

  • bun test (1487 pass), typecheck, lint, format all clean
  • bun run build + smoke test: node dist/index.js project dev --help shows --no-traces
  • protobuf decoding verified under all three ways we ship: bun from source, the Node npm bundle, and a compiled standalone binary

@github-actions github-actions Bot added the size/xl PR size: XL label Aug 12, 2026
@github-actions github-actions Bot added agentcore-harness-reviewing AgentCore Harness review in progress size/xl PR size: XL and removed size/xl PR size: XL labels Aug 12, 2026
@github-actions github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 12, 2026
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from b4d0349 to e2e696c Compare August 14, 2026 20:38
@github-actions github-actions Bot added size/xl PR size: XL and removed size/xl PR size: XL labels Aug 14, 2026
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from 56c20f7 to a59ebf1 Compare August 17, 2026 17:30
@codecov-commenter

codecov-commenter commented Aug 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.58065% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.07%. Comparing base (94f890c) to head (aabb87f).

Files with missing lines Patch % Lines
src/core/dev/otel/transforms.ts 97.32% 6 Missing ⚠️
src/io/httpServer.ts 90.62% 6 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##           refactor    #1980      +/-   ##
============================================
+ Coverage     97.06%   97.07%   +0.01%     
============================================
  Files           374      378       +4     
  Lines         22542    23033     +491     
============================================
+ Hits          21880    22359     +479     
- Misses          662      674      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch 2 times, most recently from cad8772 to 7e315b9 Compare August 17, 2026 17:53
Base automatically changed from feat/project-dev-handler to refactor August 17, 2026 20:46
Adds an in-process OTLP/HTTP collector (protobuf + JSON ingest, JSONL
persistence per trace) started by the dev handler unless --no-traces or
instrumentation.enableOtel is false. Spawned agents receive OTEL env
pointing at the collector; container runtimes get a host.docker.internal
endpoint, and Python CodeZip agents get sitecustomize-based
auto-instrumentation so uvicorn --reload workers stay traced.
…findings

- Partition each OTLP export batch by trace id before persistence: a batch
  routinely carries spans from several traces, and writing it whole to the
  first trace's file corrupted trace identity (get() missed every other
  trace in the batch). Files are now keyed <traceId>.otlp.jsonl, making
  get() a direct path lookup.
- Set signal-specific OTEL_EXPORTER_OTLP_TRACES_ENDPOINT/_LOGS_ENDPOINT
  (and protocols) alongside the generic variables: signal-specific values
  take SDK precedence, so a stray value from the shell or .env.local could
  silently redirect traces away from the local collector. The container
  rewrite now covers every OTLP endpoint variable.
- Build the PYTHONPATH test expectation with node:path.delimiter so it
  passes on Windows.
…n test

- bun audit failed on GHSA-8988-4f7v-96qf: otlp-transformer 0.213.0 pins
  @opentelemetry/core 2.6.0 (<2.8.0). Override to ^2.10.0, matching the
  version the telemetry sink packages already resolve.
- Build the expected tracesDirectory with path.join so the assertion holds
  on Windows separators.
@tejaskash
tejaskash force-pushed the feat/dev-otel-collector branch from 7e315b9 to aabb87f Compare August 18, 2026 18:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/xl PR size: XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants