OpenTelemetry events by vishnu-oai · Pull Request #2103 · openai/codex

vishnu-oai · 2025-08-09T22:23:58Z

Title

otel

Codex can emit OpenTelemetry log events that
describe each run: outbound API requests, streamed responses, user input,
tool-approval decisions, and the result of every tool invocation. Export is
disabled by default so local runs remain self-contained. Opt in by adding an
[otel] table and choosing an exporter.

[otel]
environment = "staging"   # defaults to "dev"
exporter = "none"          # defaults to "none"; set to otlp-http or otlp-grpc to send events
log_user_prompt = false    # defaults to false; redact prompt text unless explicitly enabled

Codex tags every exported event with service.name = "codex-cli", the CLI
version, and an env attribute so downstream collectors can distinguish
dev/staging/prod traffic. Only telemetry produced inside the codex_otel
crate—the events listed below—is forwarded to the exporter.

Event catalog

Every event shares a common set of metadata fields: event.timestamp,
conversation.id, app.version, auth_mode (when available),
user.account_id (when available), terminal.type, model, and slug.

With OTEL enabled Codex emits the following event types (in addition to the
metadata above):

codex.api_request
- cf_ray (optional)
- attempt
- duration_ms
- http.response.status_code (optional)
- error.message (failures)
codex.sse_event
- event.kind
- duration_ms
- error.message (failures)
- input_token_count (completion only)
- output_token_count (completion only)
- cached_token_count (completion only, optional)
- reasoning_token_count (completion only, optional)
- tool_token_count (completion only)
codex.user_prompt
- prompt_length
- prompt (redacted unless log_user_prompt = true)
codex.tool_decision
- tool_name
- call_id
- decision (approved, approved_for_session, denied, or abort)
- source (config or user)
codex.tool_result
- tool_name
- call_id
- arguments
- duration_ms (execution time for the tool)
- success ("true" or "false")
- output

Choosing an exporter

Set otel.exporter to control where events go:

none – leaves instrumentation active but skips exporting. This is the
default.

otlp-http – posts OTLP log records to an OTLP/HTTP collector. Specify the
endpoint, protocol, and headers your collector expects:

[otel]
exporter = { otlp-http = {
  endpoint = "https://otel.example.com/v1/logs",
  protocol = "binary",
  headers = { "x-otlp-api-key" = "${OTLP_TOKEN}" }
}}

otlp-grpc – streams OTLP log records over gRPC. Provide the endpoint and any
metadata headers:

[otel]
exporter = { otlp-grpc = {
  endpoint = "https://otel.example.com:4317",
  headers = { "x-otlp-meta" = "abc123" }
}}

If the exporter is none nothing is written anywhere; otherwise you must run or point to your
own collector. All exporters run on a background batch worker that is flushed on
shutdown.

If you build Codex from source the OTEL crate is still behind an otel feature
flag; the official prebuilt binaries ship with the feature enabled. When the
feature is disabled the telemetry hooks become no-ops so the CLI continues to
function without the extra dependencies.

Copilot

Pull Request Overview

This PR implements OpenTelemetry (OTEL) tracing integration for the Codex application. The work adds telemetry configuration, a new telemetry crate with OTLP exporters, and session span tracking to enable observability across the TUI and exec binaries.

Adds codex-telemetry crate with OTLP file, HTTP, and gRPC exporters
Integrates telemetry layers into TUI and exec binaries with configurable exporters
Implements session span tracking with graceful shutdown to ensure trace export

Reviewed Changes

Copilot reviewed 16 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
codex-rs/codex-telemetry/src/lib.rs	New telemetry crate implementing OTLP exporters and tracing layer builders
codex-rs/core/src/telemetry_init.rs	Configuration adapter between app config and telemetry crate
codex-rs/core/src/config_types.rs	Telemetry configuration types for TOML deserialization
codex-rs/core/src/config.rs	Integration of telemetry config with defaults and test updates
codex-rs/tui/src/lib.rs	TUI binary telemetry integration with conditional layer composition
codex-rs/exec/src/lib.rs	Exec binary telemetry integration and graceful shutdown handling
codex-rs/core/src/codex.rs	Session span tracking with proper span lifecycle management

Comments suppressed due to low confidence (1)

codex-rs/codex-telemetry/src/lib.rs:20

[nitpick] Using PathBuf::new() creates an empty path which will be ignored according to the comment. Consider using a more explicit type like Option or a dedicated enum variant to make this intent clearer.

    use opentelemetry_sdk::export::trace::SpanData;

regismesquita · 2025-09-04T19:13:39Z

Hi @vishnu-oai, this PR has been quiet for a while.
I’m preparing a broader rollout that requires OTEL support, is this branch functional?

Is there any plan at OpenAI to prioritise and merge it?

Thanks for any update you can share.

apanasenko-oai · 2025-09-06T00:51:45Z

Hi @vishnu-oai, this PR has been quiet for a while. I’m preparing a broader rollout that requires OTEL support, is this branch functional?

Is there any plan at OpenAI to prioritise and merge it?

Thanks for any update you can share.

Hi @regismesquita ,

We are working on this and next week will make more progress. Stay tuned.

Thanks

pakrym-oai · 2025-09-25T01:48:32Z

+
+    if let Some(provider) = otel.as_ref() {
+        let otel_layer = OpenTelemetryTracingBridge::new(&provider.logger).with_filter(
+            tracing_subscriber::filter::filter_fn(codex_core::otel_init::codex_export_filter),


can build_provider also return a layer with filter applied or have a method to update tracing_subscriber::registry ? so we don't spread duplicates of registration code and keep it encapsulated?

with_filter produce very complicated return type, I can try to come up with something later in follow up.

pakrym-oai · 2025-09-25T01:49:49Z

+#[cfg(feature = "otel")]
+pub mod otel_provider;
+
+#[cfg(not(feature = "otel"))]


why do we have a feature for this?

Looks like we always enable it.

some people might want to build cli without otel, but I can drop this optionality.

pakrym-oai · 2025-09-25T01:55:55Z

@@ -89,7 +109,8 @@ async fn run_stream(sse_body: &str) -> Vec<ResponseEvent> {
    while let Some(event) = stream.next().await {
        match event {
            Ok(ev) => events.push(ev),
-            Err(e) => panic!("stream event error: {e}"),
+            // We still collect the error to exercise telemetry and complete the task.
+            Err(_e) => break,


should the containing method return Result?

it will return results, here we breaking while loop of whiting on stream of events

jif-oai

More things to say on the PR but the general comment is why creating everything directly on OTEL. I feel like using tracing::instrument and maybe some prometheuse metrics (not 100% on this one, maybe the current OTEL is better but worth it to try) would be way simpler and handle a lot of things for us

jif-oai · 2025-09-25T04:34:08Z

-                user_explicitly_approved_this_action: false,
-            })
-        }
+        SafetyCheck::AutoApprove {


What's the link with this PR ?

Sorry, what do you mean?

apanasenko-oai · 2025-09-25T17:57:53Z

More things to say on the PR but the general comment is why creating everything directly on OTEL. I feel like using tracing::instrument and maybe some prometheuse metrics (not 100% on this one, maybe the current OTEL is better but worth it to try) would be way simpler and handle a lot of things for us

While tracing::instrument is indeed simpler to apply (I even used it in a prior commit), this change is more focused on event logging than on tracing code execution. The goal is to keep these events stable and treat them as part of a contract. By handling logging manually, we gain finer control over what gets logged and when.

pakrym-oai

Thank you

…e codex.session

bolinfest · 2025-10-08T06:39:19Z

-                                debug!("failed to parse ErrorResponse: {e}");
+                                let error = format!("failed to parse ErrorResponse: {e}");
+                                debug!(error);
+                                response_error = Some(CodexErr::Stream(error, None))


Why was this change made to set response_error in this case?

to catch those errors and report them as error events.

vishnu-oai requested a review from Copilot August 11, 2025 17:05

Copilot AI reviewed Aug 11, 2025

View reviewed changes

Comment thread codex-rs/codex-telemetry/src/lib.rs Outdated

Comment thread codex-rs/codex-telemetry/src/lib.rs Outdated

Comment thread codex-rs/core/src/config.rs Outdated

Comment thread codex-rs/exec/src/lib.rs Outdated

vishnu-oai requested a review from bolinfest August 11, 2025 17:44

vishnu-oai marked this pull request as ready for review August 11, 2025 17:44

vishnu-oai changed the title ~~[wip] OTEL tracing~~ OpenTelemetry tracing Aug 11, 2025

vishnu-oai requested a review from dylan-hurd-oai August 12, 2025 10:00

vishnu-oai force-pushed the otel-tracing branch from 0c1fffd to 4cae7be Compare September 5, 2025 15:31

bolinfest requested review from apanasenko-oai and removed request for bolinfest September 10, 2025 17:51

apanasenko-oai requested review from pakrym-oai and removed request for apanasenko-oai September 10, 2025 20:20

apanasenko-oai force-pushed the otel-tracing branch 6 times, most recently from 5606d4c to 8d3967a Compare September 13, 2025 00:24

apanasenko-oai changed the title ~~OpenTelemetry tracing~~ OpenTelemetry events Sep 13, 2025

apanasenko-oai force-pushed the otel-tracing branch 8 times, most recently from d74f8b6 to 9d3e066 Compare September 15, 2025 21:50

pakrym-oai reviewed Sep 16, 2025

View reviewed changes

Comment thread docs/config.md

pakrym-oai reviewed Sep 16, 2025

View reviewed changes

Comment thread docs/config.md Outdated