[OTel] Chatty per-event Responses spans dominated native trace volume - 49% across 5 real-work sessions

[zhichli]

What version of Codex CLI is running?

Reproduced with codex-cli 0.146.0; the same instrumentation is present on current main (4642370542739d5dd080b0c87a9de06a6435d3db). My current standalone install is 0.146.1.

What subscription do you have?

Not relevant to the native telemetry instrumentation path; these runs used codex exec through a configured Responses provider.

Which model were you using?

gpt-5.6-sol

What platform is your computer?

Linux 6.19.8-surface-3 x86_64 x86_64

What terminal emulator and version are you using (if applicable)?

Not relevant; the measurements came from native OTLP export during non-interactive codex exec runs.

Codex doctor report

Not included because this is an instrumentation-cardinality issue rather than an installation failure. The current report completes with overallStatus: ok on Codex 0.146.1.

What issue are you seeing?

I've been consuming OTel data from both Codex CLI and Codex in VS Code lately, but the native traces have been difficult to use because a large part of each waterfall is routine stream bookkeeping. I dug a bit deeper and ran five real read-only research workloads with native OTLP logs, traces, and metrics enabled.

I'm not trying to nitpick every short or internal span—some of them are useful—but one pair stands out: every Responses stream event creates a handle_responses span and a child receiving span.

Across the five runs:

Measurement	Result
Model sampling requests	147
Total exported spans	185,982
handle_responses spans	45,959
receiving spans	45,959
Combined share of all spans	49.42%
Average Responses events per sampling request	312.6
handle_responses spans with neither from nor usage fields	98.3%

The volume scales with protocol fragmentation rather than meaningful operations: 147 model requests produced 91,918 spans from this pair alone.

The receiving child mostly measures the wait for the next stream event, while most handle_responses spans represent routine text, reasoning, or tool-input deltas. Codex already exports lower-cardinality signals for the useful parts of this flow: a receiving_stream span, SSE counters and duration histograms, API/request timing, TTFT, completion usage, output-item boundaries, tool calls, and errors.

The current implementation creates both spans inside the event loop:

• handle_responses and receiving
• the native trace layer exports every tracing span

This seems similar in spirit to #29432, which removed per-event WebSocket logs/OTel events while preserving counters, duration metrics, response timing, parsing, and errors.

What steps can reproduce the bug?

1. Configure native trace export:

   [otel]
   trace_exporter = { otlp-http = {
       endpoint = "http://127.0.0.1:4318/v1/traces",
       protocol = "json"
   } }

2. Run a normal codex exec task that produces a streamed response. Tool-heavy or research tasks make the amplification easier to see because they use multiple sampling requests.

3. Group the exported spans by name.

4. Compare the number of run_sampling_request/receiving_stream spans with handle_responses and receiving.

A representative aggregate from my five runs was:

run_sampling_request   147
handle_responses       45,959
receiving              45,959

The runs were concurrent, so I am not using their durations as a latency benchmark. The span counts and per-run attributes were isolated with unique benchmark IDs.

What is the expected behavior?

Native trace cardinality should stay reasonably proportional to meaningful operations rather than the number of text/reasoning/tool-input deltas.

I would keep:

• one receiving_stream span per sampling request;
• output-item and completion boundaries;
• completion token usage;
• model/API request and tool-call spans;
• TTFT, SSE counters, and duration histograms;
• parsing, transport, and response failures.

My preference would be to avoid a full span for each routine delta. The patch below demonstrates one bounded tradeoff: retain semantic boundaries while suppressing routine delta spans. If the per-event wait timing is important diagnostically, I would be happy to adjust the representation—for example, preserving it with a histogram, maximum-gap attribute, or thresholded slow-gap event.

Additional information

I prepared a small proposed fix to make that tradeoff concrete. It:

• removes the per-event receiving child span;
• skips handle_responses spans for routine text, reasoning-summary, reasoning-content, and tool-input deltas;
• preserves handle_responses spans for semantic boundaries such as output items and completion;
• keeps the existing SSE counters/duration metrics and adds regression coverage for the reduced span set.

Proposed diff in my fork: zhichli:fix/bound-responses-event-spans

Validation performed:

• just test -p codex-core record_responses_sets_span_fields_for_response_events
• just test -p codex-core handle_responses_span_records_response_kind_and_tool_name
• just test -p codex-core turn_and_completed_response_spans_record_token_usage
• just fix -p codex-core
• just fmt

This is intentionally limited to Responses-event spans. Rollout-persistence and detached auth spans have separate tradeoffs and should not be bundled into this change.

This is my first Codex issue. If the diagnosis and proposed direction make sense, I'm happy to contribute the patch if invited.

AI assistance: I used AI assistance for parts of the investigation, benchmark analysis, patch, and issue draft. I reviewed the source changes, measurements, links, and test results.