[Discussion][Observability 1/2] Recording Agent Traces in the Event Log

[joeyutong]

This proposal is the first in a two-part Flink Agents observability series:

1. Recording Agent Traces in the Event Log (this proposal)
2. Built-in Operational Metrics for Flink Agents (follow-up)

The two proposals serve complementary purposes. Agent Trace reconstructs what happened during a single run, while operational metrics describe the aggregate health and behavior of many runs.

1. Context and Existing Work

Flink Agents currently provides three main observability surfaces: metrics, Event Log, and EventListener. They provide aggregate signals, persisted business Event records, and callback hooks, but do not capture enough context to reconstruct an agent run as a causal trace.

This proposal builds on the gap identified in Discussion #710 and focuses on recording. Related community work includes:

Community work	Existing progress	Relationship to this proposal
Discussion #710	Identifies the lack of per-run causal traces and separates recording from reconstruction and visualization.	Provides the overall problem statement; this proposal addresses the recording side.
Issue #841	Proposes adding run identity, the emitting Action, and source Event information to existing Event Log records.	Overlaps with this proposal on business Event lineage. This proposal also introduces Execution Events for Action, LLM, Parser, and Tool execution boundaries.
Discussion #876 and PR #887	Extend the business Event taxonomy with typed Memory Events and an AgentRunBeginEvent carrying a short-term-memory snapshot.	This proposal retains them as business Events and applies the same trace context to them as to other logged Events.

To close the remaining recording gap, this proposal combines two complementary forms of context:

Agent Trace = business Event causality + execution hierarchy

2. Motivating Example

Consider an Action that calls an LLM and parses the response into a structured output:

InputEvent
  -> chat_model_action
       -> LLM call: success
       -> structured-output parser: failed

The Event Log may contain the original input, a chat request, a chat response, and an Action failure. Reconstructing the complete run requires both lineage and execution information.

Run identity and business Event lineage answer:

• Which records belong to the same run?
• Which Event triggered an Action, and which Action emitted a later Event?

Execution information answers:

• Was the failure caused by the model call or by parsing its response?
• Which concrete invocation of chat_model_action contained those operations?
• How long did the LLM call and Parser execution take independently?

Event lineage alone cannot represent the LLM and Parser boundaries because those operations are not business Events. A complete trace therefore needs both forms of context.

3. Goals and Non-goals

Goals

• Associate Event Log records with one run, business key, and Agent.
• Reconstruct the actual Event-to-Action causal relationship at runtime.
• Represent Action, LLM, Parser, and Tool execution boundaries.
• Reconstruct parent-child relationships between nested executions.
• Record execution outcomes and stable failure categories.
• Preserve the original business Event payload.
• Keep Java and Python Event Log records semantically aligned.
• Extend the existing Event Log instead of introducing an independent tracing backend.

Non-goals

• Trace query APIs or storage indexing.
• Trace reconstruction or visualization.
• OpenTelemetry export.
• Evaluation datasets or experiment tracking.
• Dashboards, alerts, or other product-level capabilities.
• Provider-attempt-level tracing for retries.
• Session-level tracing across multiple inputs.

4. Proposed Trace Model

The existing Event Log model records a business Event together with its occurrence context:

Existing EventLogRecord
  = Event
  + EventContext

This model records which business Event occurred and when it was processed. It does not identify the run or execution to which the record belongs, and it cannot represent runtime work that is not a business Event.

The proposed model extends the record as follows:

Proposed EventLogRecord
  = Event or Execution Event
  + EventContext
  + ExecutionTraceContext

The proposal leaves the responsibilities of Event and EventContext unchanged. It adds Execution Events for runtime lifecycle boundaries and ExecutionTraceContext for run identity, Event-to-Action causality, and execution hierarchy. Together, business Events describe workflow causality, while Execution Events describe the runtime work performed at each Action node.

Abstraction	Responsibility
Event	Describes what happened through an Event id, type, and attributes.
Execution Event	Describes the lifecycle and result of one execution.
EventContext	Carries occurrence metadata, currently the Event type and timestamp.
ExecutionTraceContext	Associates the record with a run, Action scope, execution hierarchy, and execution entity.
EventLogRecord	Combines an Event, its occurrence context, and its trace context into one persistent record.

4.1 Business Event and Execution Event

Business Events and Execution Events have distinct semantics:

• A business Event represents a workflow occurrence and retains its original payload. It continues to participate in Event routing, EventListener callbacks, and existing business Event metrics.
• An Execution Event represents the lifecycle of one execution. It records started, finished, failed, or reused and is used only for runtime observation.

An Execution Event is a runtime recording abstraction, not a new public Event base class. It reuses the existing Event data structure, but does not enter EventRouter, notify EventListener, trigger Actions, or contribute to business Event metrics.

Trace information remains outside user-owned Event attributes. A typed business Event, including a Memory Event or AgentRunBeginEvent, can receive the same run and execution context when written without changing its payload schema.

4.2 Trace Relationships

In this proposal, a run is one processing instance initiated by an input for one business key. It is not a user session and does not span multiple inputs. The examples use input_run_id to make this scope explicit; the final name can be aligned with the existing Agent Run terminology.

An execution is one concrete invocation of an Action, LLM, Parser, or Tool. Invoking the same entity twice within one run creates two distinct executions.

The trace combines two relationships:

1. Business Events connect Action executions into a causal graph.
2. Parent execution identifiers connect operations performed inside an Action into an execution tree.

flowchart TB
    IN["Input Event"] -->|"triggers"| A1["Action execution A"]
    A1 -->|"contains"| L1["LLM execution"]
    A1 -->|"contains"| P1["Parser execution"]
    A1 -->|"emits"| E1["Business Event E"]
    E1 -->|"triggers"| A2["Action execution B"]
    A2 -->|"contains"| T1["Tool execution 1"]
    A2 -->|"contains"| T2["Tool execution 2"]

Loading

For business Event causality:

• An Action execution records the business Event occurrence that triggered it.
• Action executions connected by business Events are siblings rather than parent and child executions.
• Static Action trigger rules describe possible routes, but cannot identify the Event occurrence that triggered a concrete Action execution under concurrency, retry, or replay.

For execution hierarchy:

• The run groups all trace records and serves as the trace root; it is not itself an execution.
• LLM, Parser, and Tool executions are children of the Action execution in which they run.

Together, business Event records reconstruct the workflow graph, while Execution Events and their parent identifiers reconstruct the work performed within each Action execution.

5. Event Log Record

The serialized representation remains a flat JSONL record so that commonly queried fields can be indexed and aggregated without parsing nested envelopes. The in-memory model preserves the separation between Event, EventContext, and ExecutionTraceContext.

Field group	Fields	Source	Purpose
Event occurrence context	timestamp	EventContext	Records when the Event occurrence was processed.
Event	event_id, event_type, event_attributes	Event or Execution Event	Preserve the Event identity, semantics, and payload.
Run identity	input_run_id, business_key, agent_name	ExecutionTraceContext	Group records from one input-processing run.
Execution hierarchy	execution_id, parent_execution_id, action_trigger_event_id	ExecutionTraceContext	Reconstruct nested executions and Event-to-Action causality.
Execution entity	entity_type, entity_name, entity_metadata	ExecutionTraceContext	Identify the primary execution entity and carry its supplemental metadata.
Execution outcome	status, problem_category	Execution Event attributes	Describe the lifecycle result and stable failure category of an Execution Event.

EventContext mirrors Event.getType() as part of its occurrence contract, and the serializer writes that value once as event_type. For Execution Events, status and problem_category are promoted from Event attributes to top-level fields during serialization.

entity_type and entity_name identify the primary execution node, such as an Action, LLM, Parser, or Tool invocation. entity_metadata contains supplemental attributes for that node; it neither introduces another execution node nor defines a parent-child relationship.

parent_execution_id and action_trigger_event_id capture different relationships:

• parent_execution_id represents execution containment.
• action_trigger_event_id identifies the business Event occurrence that triggered the current Action scope.

Child LLM, Parser, and Tool records may copy the Action's trigger Event id so that the Action scope can be queried directly. This does not imply that the business Event directly invoked the child execution.

For example, a structured-output parsing failure may be written as:

{
  "timestamp": "2026-07-02T01:23:45.678Z",
  "input_run_id": "run-1",
  "business_key": "order-1",
  "agent_name": "order-agent",
  "execution_id": "parser-execution-1",
  "parent_execution_id": "action-execution-1",
  "action_trigger_event_id": "input-event-1",
  "entity_type": "parser",
  "entity_name": "structured_output",
  "event_id": "event-17",
  "event_type": "_execution_failed_event",
  "status": "failed",
  "problem_category": "model_output_parse_error",
  "event_attributes": {
    "error_type": "JsonProcessingException",
    "error_message": "Unable to parse the model response"
  }
}

event_id identifies a logical Event occurrence; it does not imply that the corresponding physical log record is written exactly once.

6. Recording Boundaries

Run Context

When an input enters the Agent runtime, the framework creates or restores a run context containing:

• input_run_id
• business_key
• agent_name

The same context is propagated to business Events and executions created during the run.

Action

The runtime records one Action execution for each concrete invocation:

• started before invoking the Action.
• finished after successful completion.
• failed when an unhandled Action exception terminates the invocation.
• reused when ActionStateStore returns an already completed Action result without re-executing the Action.

Action-level failures use action_execution_failed; the runtime does not infer a more specific category from a child failure. LLM, Parser, and Tool executions record their own precise failure categories.

LLM

An LLM execution represents one logical ChatModel call rather than an individual provider attempt. In the initial scope, intermediate retry attempts are not separate executions.

• A final model response records success.
• A final model-call failure records model_call_failed.

Parser

Structured-output parsing has its own Parser execution so that consumers can distinguish between:

• A model call that failed.
• A model call that succeeded but returned an unparseable response.

Parser failures use model_output_parse_error. Recording the Parser boundary must not change the existing retry or ignore behavior of structured-output processing.

Tool

A Tool execution represents one concrete tool invocation. If a ToolRequestEvent requests multiple calls, each invocation produces a separate Tool execution.

• A successful Tool response records success.
• A Tool error response keeps the original ToolResponseEvent payload but records the Tool execution as failed.
• A thrown Tool exception records tool_call_failed and preserves the underlying error type and message.

Tool calls, MCP Servers, and Skills have different roles in the trace model:

Concept	Execution node	Trace representation
Tool call	Yes	Each invocation is one Tool execution.
MCP Tool call	Yes, as a Tool execution	The MCP Server is the resource exposing the Tool, not another execution level.
MCP Server	No	Recorded as resource provenance on its MCP Tool executions.
Skill	No	Represents instructions and resources; only an explicit load_skill invocation is an execution.

For example, an MCP Tool call has one execution node:

Action execution
  -> Tool execution: search
       metadata:
       - tool type: MCP
       - MCP Server: search_server

In the serialized record, entity_type and entity_name identify the Tool execution as the primary entity. Its Tool kind and MCP Server provenance are stored in entity_metadata, not represented as child execution nodes:

{
  "entity_type": "tool",
  "entity_name": "search",
  "entity_metadata": {
    "tool_type": "mcp",
    "mcp_server": "search_server"
  }
}

The MCP Server therefore has no separate execution id or lifecycle Events. Its metadata can still be used to query or aggregate related MCP Tool executions.

load_skill is a built-in Tool that reads the requested Skill's SKILL.md content or a specific resource and returns it to the model. Its invocation is represented as a Tool execution, with the Skill name and optional resource path stored as metadata. The runtime does not propagate a Skill execution context to subsequent Tool calls, so the trace does not attribute those calls to the loaded Skill.

Retry and Replay

Retry and replay do not have dedicated Event types. Depending on the runtime path, a repeated operation appears as a new execution or another physical Event Log record.

7. Runtime Integration and Compatibility

Business Events and Execution Events follow separate semantic paths but share the same Event Log writer:

flowchart TD
    BE["Business Event"] --> ER["EventRouter"]
    ER --> EL["EventListener"]
    ER --> BM["Business Event metrics"]
    ER --> AR["Action routing"]
    ER --> W["Event Log writer"]

    EE["Execution Event"] --> ES["ExecutionEventSink"]
    ES --> W
    W --> LOG["Event Log"]

Loading

This separation preserves existing externally observable behavior:

• Business Events retain their existing EventRouter behavior.
• Execution Events do not trigger Actions or notify user EventListener implementations.
• Both paths use the same writer lifecycle and output format.
• Event Log writes remain best-effort and do not affect the outcome of an Action.
• Existing EventLogger implementations can continue to handle Event and EventContext without consuming trace context.

New records use a normalized, flat JSON format with snake_case field names. The deserializer continues to support the previous nested format. Legacy records have no run or execution context and therefore cannot form a complete trace.

Explicit execution boundaries add lifecycle records, so log volume grows with the number of recorded Action, LLM, Parser, and Tool executions. The proposal uses the existing Event Log path instead of introducing a separate tracing exporter or recording pipeline.

8. Recovery and Replay Semantics

The Event Log is best-effort and does not provide exactly-once delivery.

Without ActionStateStore:

• Source replay may process the same logical business input again.
• The replayed input currently receives new run and execution identities.
• Each resulting run still has an internally consistent execution hierarchy.

With ActionStateStore:

• A completed Action can be reused without re-executing user code.
• The current run records a reused Execution Event for that Action.
• Persisted output Events continue through the current run.
• The original ExecutionTraceContext is not stored with the completed Action, so the Event Log cannot link a replayed output Event to the execution that originally produced it.

Consumers must not infer missing identities from timestamps or payload content.

9. Prototype Status

The model has been prototyped across the Java and Python runtime paths. The prototype validates:

• A shared Event Log schema for business Events and Execution Events.
• Action, LLM, Parser, and Tool lifecycle records, including success and failure paths.
• Event-to-Action causality, nested executions, and multiple Tool calls from one request.
• MCP Server and explicit Skill metadata.
• Java and Python semantic alignment through conformance tests and end-to-end probes.

The purpose of this discussion is to review the model and contracts before submitting the implementation upstream.

10. Summary

Business Event lineage is necessary for Agent Trace, but it is not sufficient. A complete trace must also represent runtime executions that are not business Events.

This proposal extends the existing Event Log with:

• Run identity and business key association.
• Explicit Event-to-Action causality.
• Action, LLM, Parser, and Tool execution lifecycle records.
• Parent-child execution relationships.
• Stable execution outcomes and failure categories.
• A shared recording format for Java and Python.

The result is a framework-native recording model that allows consumers to reconstruct a run without changing business Event payloads or introducing a separate tracing system.