[Discuss][Observability] OpenTelemetry GenAI export for Agent Traces (follow-up to #900)

[Zhuoxi2000]

Motivation

#900 defines how Agent Traces are recorded in the Event Log, and deliberately lists OTel export as a non-goal of the recording scope. In that thread, @joeyutong agreed that OpenTelemetry export should be a supported follow-up once the recording contract settles, with the exact GenAI attribute mapping to be worked out as part of the exporter design. This discussion is that follow-up: a config-gated exporter that maps recorded Execution Events to the OpenTelemetry GenAI semantic conventions, so any OTLP-compatible backend (Jaeger, Grafana Tempo, Langfuse, Phoenix, vendor APMs) can visualize and analyze agent runs without flink-agents shipping its own UI.

Proposed mapping

Trace topology, aligned with the #900 recording model:

flink-agents	OTel GenAI
Input run (input_run_id)	Trace (one run = one trace)
Execution (execution_id / parent_execution_id)	Span / parent Span
Agent execution	span invoke_agent {agent_name}; gen_ai.operation.name = invoke_agent, gen_ai.agent.name
Tool execution	span execute_tool {tool_name}; gen_ai.operation.name = execute_tool, gen_ai.tool.name
Model call	span chat {model}; gen_ai.operation.name = chat, gen_ai.request.model, gen_ai.provider.name
business_key	gen_ai.conversation.id
Lifecycle events (started/finished)	span start/end timestamps
status / problem_category	span status + error.type
Token usage	gen_ai.usage.input_tokens / gen_ai.usage.output_tokens

ID derivation. Framework IDs stay opaque, as preferred in #900; the exporter derives fixed-width OTel IDs deterministically, e.g. trace_id = truncate(SHA-256(input_run_id), 128 bits) and span_id = truncate(SHA-256(execution_id), 64 bits). Deterministic derivation keeps re-exports idempotent.

Design points to agree on

1. Where the exporter runs. Two candidate shapes:

   • (a) In-process: an Execution Event listener in the runtime feeds an async OTLP exporter (batching, bounded queue, drop-on-overflow so it can never backpressure the job).
   • (b) Out-of-band: a separate consumer reads the Event Log and exports — zero impact on the running job, works retroactively, but requires event-log trace recording ([api][runtime][python] Add Agent Trace recording to Event Log #924) to be enabled.

   I lean toward (b) first, since it derives everything from the recording contract and adds no runtime risk, keeping (a) as a later option. Opinions welcome.

2. Config surface. event-log.trace.otel.enabled (default false, mirroring the recording flag), OTLP endpoint/protocol/headers, service.name. Content capture (prompt/response bodies) stays off by default and is opted into separately — gen_ai content capture has privacy and volume implications.

3. Semconv version. The gen_ai.* conventions are still development stability; we should pin the exported schema_url and document the target version. I'm engaging in the OTel GenAI SIG, so we can track upstream changes as they stabilize.

4. Scope for 0.4. Phase 1 = Java, spans-only, out-of-band exporter + config; Phase 2 = usage-metrics enrichment; Phase 3 = Python parity. Is Phase 1 realistic for the 0.4 feature freeze (Sept 15)?

What I'd deliver

I've already implemented the inverse mapping (OTel GenAI → storage) in a side project, so the attribute table above is grounded in working code. If maintainers agree with the general shape, I'll break this into concrete issues and start with Phase 1.