[api][runtime][python] Add Agent Trace recording to Event Log

[joeyutong]

Linked discussion: #900

Related: #710, #841, #923

Purpose of change

This PR implements the recording side of Agent Trace proposed in #900.

The existing Event Log records business Events but does not provide enough runtime context to identify one input run, distinguish concrete Action/LLM/Parser/Tool executions, reconstruct nested execution relationships, or attribute execution failures.

This PR adds execution identity and lifecycle recording to the existing Event Log path. It complements the business Event lineage implemented in #923 rather than defining another Event-to-Action lineage field:

• Business Event lineage remains responsible for cross-Action causality.
• This PR records concrete execution identity and parent-child containment.
• Business Events emitted by an Action retain that Action execution's executionId, connecting the two models.

API and trace model

• ExecutionTraceContext carries run identity, execution hierarchy, entity identity, and entity metadata.
• ExecutionLifecycleEvents defines started, finished, failed, and reused lifecycle Events.
• ExecutionReporter and ExecutionReporters provide an optional, best-effort capability for reporting nested executions.
• ToolExecutionMetadataProvider allows Tools to contribute small structured execution metadata.
• EventLogger accepts an optional ExecutionTraceContext; its default overload preserves compatibility with existing implementations.
• AgentPlan carries the Agent name used by trace records.
• event-log.trace.enabled controls Trace persistence and defaults to false.

Runtime integration

• ActionExecutionOperator creates one run context per processed input and one execution context per Action invocation.
• ActionTask carries the Action execution context and started-event marker across continuations and Flink state restoration.
• ActionTaskContextManager keeps child-execution start/terminal pairing transient and scoped by Action execution id across live continuation tasks.
• Action executions emit lifecycle Events around invocation and emit reused when completed Action state is reused.
• Business Events continue through EventRouter.
• Built-in lifecycle reporting sends Execution Events through ExecutionEventSink without automatically submitting them to the business Event routing path.
• RunnerContextImpl implements ExecutionReporter, creates child execution contexts, and pairs start and terminal reports.
• ExecutionEventLogger sends Execution Events to the shared EventLogWriter.
• EventLogWriter owns logger open, append, flush, and close operations; write failures remain best-effort.

Action and resource instrumentation

• ChatModelAction records one LLM execution per framework model invocation.
• Structured-output parsing is recorded as a separate Parser execution.
• ToolCallAction records one Tool execution per concrete tool call, including error responses and thrown exceptions.
• The instrumentation does not change existing retry, ignore, or Tool-call execution semantics.
• MCP Tools remain Tool executions; MCP Server identity is stored as metadata.
• Explicit load_skill calls record the loaded Skill as Tool execution metadata.
• Failure Events preserve the deepest available cause and use stable problem categories.

Python alignment

• Python exposes the same ExecutionReporter capability and entity/problem vocabulary as Java.
• Python LLM, Parser, and Tool boundaries report the same lifecycle semantics.
• Python reporting delegates to the runtime context without introducing a second trace model.
• Python Event IDs represent unique Event occurrences rather than content-derived identities.

Event Log format and compatibility

EventLogRecord combines EventContext, optional ExecutionTraceContext, and Event.

The JSONL representation is flattened for querying and aggregation. Framework-owned fields retain the existing camelCase convention, including eventType, eventAttributes, inputRunId, and executionId. The record includes Event occurrence fields, run identity, execution hierarchy, entity information, lifecycle status, failure category, and Event attributes.

The framework deserializer continues to read the previous nested Event Log format. Existing external consumers that parse the raw JSON shape must migrate to the normalized field names.

Trace persistence is disabled by default. When disabled, business Events continue to be logged without trace context and Execution Events are not persisted.

A restored ActionTask retains its run and execution identities. A source-replayed input not represented by restored Action state starts a new run.

This PR changes the pending ActionTask state schema. Restoring savepoints containing that state from before this change would require a versioned ActionTask state serializer, which is outside this PR.

Tests

• Added and updated Java tests for trace contexts, lifecycle Events, Event Log serde, Event Log writing, Action execution, retries, Tool reporting, MCP metadata, and Skill metadata.
• Added and updated Python tests for reporter behavior, LLM/Parser retries, Tool reporting, MCP metadata, and Skill metadata.
• mvn -pl runtime -am -DskipITs test
  • API: 283 tests passed, 9 skipped.
  • MCP integration: 39 tests passed.
  • Plan: 199 tests passed, 1 skipped.
  • Runtime: 425 tests passed.
• Focused Java validation for lifecycle reporting, Event Log serde/writing, Action execution, Tool/MCP/Skill metadata: 105 tests passed.
• Focused Python validation for reporter, Tool, MCP, and Skill paths: 35 tests passed.
• Java Spotless and Python Ruff checks passed.

API

This PR introduces the public Trace APIs described above and extends EventLogger with a backward-compatible default overload accepting optional trace context.

The normalized raw Event Log JSON shape is a compatibility change for external consumers. Legacy records remain readable through the framework deserializer.

Documentation

• doc-needed
• doc-not-needed
• doc-included

[weiqingy]

Thanks for taking this on. One thing worth surfacing: the Python Event.id switch to a per-occurrence uuid4 also fixes a real collision. chat_model_action.py keys sensory-memory dicts on the request event's id, so two identical ChatRequestEvents on the same key used to collapse into one and splice their conversations together. The PR body frames it as identity-semantics alignment, so you may want to update the description to match what the code actually does.

A few questions inline, plus one below that spans files.

The description mentions this complements the Event lineage work in #923. Going through both diffs, I think there may be an interaction worth checking before either lands.

The new serializer writes the Event's attributes map rather than the Event itself: EventLogRecordJsonSerializer.java:86 emits eventAttributes via the helper at :90-91, where the version it replaces did mapper.valueToTree(event). #923 adds upstreamEventId and upstreamActionName as fields on Event with getters, not as attribute entries.

If I'm reading that right, those two fields wouldn't reach the Event Log once the flat shape lands, which is the artifact #841 set out to populate. The same applies to tools/reconstruct_trace_tree.py, which reads record["event"] and would hit a KeyError with no event key present. Nothing in a merge would surface either one, since Event.java isn't touched here.

More generally, should framework-owned metadata on Event have a home in the flat record? upstreamEventId and upstreamActionName read like the same family as executionId, so a top-level slot would seem consistent, but is there a reason to keep them out?

[weiqingy]

started here and succeeded at :389 bracket ctx.durableExecute(...), but on replay that returns the cached result and short-circuits (RunnerContextImpl.java:383-385) while the action body still re-runs. Your own testEarlierCheckpointReplayKeepsDurableState:1039 pins that: DURABLE_CALL_COUNTER stays at 1 while the output is produced again.

So after a fine-grained recovery the log gets a complete started plus succeeded pair, with a fresh executionId, for a model call that never happened. ToolCallAction has the same shape. Since counting LLM calls is the first thing anyone does with this log, cost dashboards would over-report spend and under-report latency.

What makes me read this as unintended: the Action level already has ExecutionLifecycleEvents.executionReused() for exactly this case, but nothing below Action can emit it, since ExecutionReporter has no such method.

Could a cached durable result surface a reused signal that the reporters emit instead of started/succeeded? Or does the reporting want to move inside the durable boundary? And if it's out of scope here, would naming it in the monitoring doc be enough for now?

[weiqingy]

Passing rootNode.get("eventAttributes") into truncator.truncate(...) narrows truncation in two ways. Same block at Slf4jEventLogger.java:179-181, so both loggers are affected.

The protections now point at the wrong names. JsonTruncator is untouched by this PR and still skips PROTECTED_FIELDS = {eventType, id, attributes} at isTopLevel (:55-56, :117-119). Those were envelope names; under the flat shape they are user attribute names, so an attribute called id escapes event-log.standard.max-string-length at STANDARD. JsonTruncatorTest.testProtectedFields:136-161 still pins the old shape and passes, so CI won't catch it. Worth noting #923 adds upstreamEventId and upstreamActionName to that same set, which won't take effect under the flat shape either.

Separately, entityMetadata sits as a sibling of eventAttributes (EventLogRecordJsonSerializer.java:73), so the truncator never sees it. It's fed by the public ToolExecutionMetadataProvider hook, and LoadSkillTool.java:68-80 copies the model-supplied name and path straight in, so an LLM-controlled value lands unbounded.

Would running the truncator over the whole record, with the new framework field names protected, close both at once? That was my first instinct, though I may be missing why the narrowing was deliberate. Either way, would testProtectedFields want re-pointing at what the production path now passes in?

[weiqingy]

Anchoring here since this is where the contract gets pinned: chatReportsEachRetriedModelInvocation asserts times(2) on reportExecutionStarted. started sits inside the retry loop (ChatModelAction.java:371-373), and Python matches (chat_model_action.py:352-375), so both languages emit one execution per attempt.

On the design discussion the contract landed on one execution per framework model invocation, with retries collapsing into it and Parser recorded separately. Parser is implemented that way (:457-474); the retry half went the other way.

Downstream, a retryCount of 3 on a successful request reads as 4 LLM executions at a 25% success rate, and nothing in the record separates "attempt 2" from "a second call". ChatResponseEvent already carries retryCount, so the two views disagree.

Which way did you land in the end? Two shapes if either helps: hoist started/succeeded outside the loop and put attempts on the single terminal, or keep per-attempt and add an ordinal plus a shared correlation id. Either way, would it be worth pinning in ExecutionReporter's javadoc, now that it's permanent public API?