[Discussion][Observability 2/2] Built-in Operational Metrics for Flink Agents

[joeyutong]

This proposal is the second in a two-part Flink Agents observability series:

1. Recording Agent Traces in the Event Log
2. Built-in Operational Metrics for Flink Agents (this proposal)

Part 1 defines the per-run recording needed to reconstruct an Agent Trace. This proposal defines bounded-cardinality metrics for observing aggregate health and behavior across runs.

The two capabilities share runtime observation boundaries but produce independent outputs:

Agent Trace         = per-run recording
Operational metrics = aggregate measurement

1. Context and Relationship to Part 1

Flink already provides job, operator, checkpoint, backpressure, and resource metrics. These metrics describe the Flink execution layer, but they do not directly answer Agent-specific operational questions:

• Are input runs succeeding, and is their end-to-end latency increasing?
• Are inputs waiting behind other inputs for the same key?
• Is an Action waiting to be scheduled or spending too long on asynchronous work?
• Are failures or latency concentrated in a particular model resource, Tool, Skill, or MCP Server?
• Are model retries or token consumption increasing unexpectedly?

Part 1 proposes two runtime Event categories for per-run observation:

• Business Events describe workflow occurrences and Event-to-Action causality.
• Execution Events describe the lifecycle of Action, LLM, Parser, and Tool executions.

Those Events also provide natural boundaries for aggregate metrics. Trace logging and metric aggregation can consume the same Execution Events independently, so Action, LLM, and Tool metrics follow the lifecycle semantics recorded in each trace.

Execution Events cover only work represented as executions. Queue depth, ActionTask scheduling, token usage, and retry statistics instead arise from runtime state or call results and are recorded at their own boundaries. The complete signal model is therefore:

flowchart LR
    BE["Business Event"] --> TRACE["Event Log<br/>per-run Agent Trace"]
    BE --> EVENT_METRICS["Event-driven metrics<br/>Business Event / Action / LLM / Tool"]

    EE["Execution Event"] --> TRACE
    EE --> EVENT_METRICS

    RUN_STATE["Run and queue state"] --> RUN_METRICS["Input Run and scheduling metrics"]
    MODEL_RESULT["Model call result"] --> MODEL_METRICS["Token usage and retry metrics"]

Loading

This model aligns Agent Trace and execution metrics on lifecycle, entity, outcome, and metadata semantics while keeping the two outputs independent.

2. Goals and Non-goals

Goals

• Define built-in metrics for Input Run health, Action scheduling, and Agent execution entities.
• Align Action, LLM, and Tool metric boundaries with the Execution Event model from Part 1.
• Define stable metric names, types, scopes, and outcome semantics.
• Keep metric dimensions bounded and suitable for aggregation.
• Preserve the distinction between configured resources and provider-returned model names.
• Keep Java and Python metric semantics aligned.
• Define metric behavior across task attempts and recovery.

Non-goals

• Dashboard layout, visualization, or alert thresholds.
• Reporter-specific queries or Prometheus expressions.
• Per-run diagnosis through metric labels.
• Exactly-once metric values across task attempts.
• Provider-attempt-level LLM metrics.
• Evaluation datasets, experiments, or model-quality metrics.

3. Metric Model

3.1 Scope Hierarchy

The Agent name identifies the Flink operator that runs the Agent. Agent-specific dimensions are registered as key-value MetricGroup scopes. Metric reporters that support dimensions, such as Prometheus, can expose these scopes as labels; other reporters may encode them in the metric identifier.

Agent operator
├─ Agent-level metrics
└─ action=<action_name>
   ├─ Action-level metrics
   ├─ model_resource=<resource_name>
   ├─ tool=<tool_name>
   ├─ skill=<skill_name>
   ├─ mcp_server=<server_name>
   └─ model=<model_name>

Scope	Meaning
Agent	Aggregate behavior of the Agent operator, without selecting an Action
Action	Scheduling and execution behavior of one Action
Model Resource	The ChatModel resource name referenced by the Agent plan; used for call health and retries
Model	The model name returned by the provider; used for token consumption
Tool	One named Tool; used for Tool-call health
Skill	The Skill name carried by an explicit load_skill Tool call
MCP Server	The MCP Server name carried by an MCP Tool execution

model_resource and model belong to independent branches:

• Model Resource metrics answer whether a configured ChatModel resource is healthy.
• Model metrics attribute token consumption to the model name returned by the provider.

When one resource is shared by multiple Actions, each Action produces a separate raw metric series. Consumers can retain the Action scope to locate the caller or aggregate across that scope to view resource-wide health.

3.2 Metric Types and Naming

The proposal uses standard Flink metric types:

Type	Semantics
Counter	A cumulative count since the current task attempt started
Meter	A rate derived from a Counter
Histogram	A distribution of latency samples observed in the current task attempt
Gauge	A current value that can increase or decrease

Metric names follow the existing Flink Agents style:

• Counts use numOf....
• Rates use ...PerSec.
• Millisecond latency distributions use ...LatencyMs.
• Existing domain-specific names for token usage and retries are retained for compatibility.
• Scope keys use snake_case, such as model_resource and mcp_server.

3.3 Cardinality

Operational metrics intentionally exclude per-run and per-execution identifiers. Fields such as input_run_id, execution_id, business_key, and event_id are useful for Agent Trace queries but would create unbounded metric cardinality.

Metric scopes are limited to reusable Agent entities and resources: Action, Model Resource, model name, Tool, Skill, and MCP Server. Metrics identify where aggregate behavior is concentrated; Agent Trace remains the per-run diagnostic surface.

4. Built-in Metrics

The existing Event throughput, Action throughput, and token metrics remain part of the built-in metric surface. This proposal adds Input Run, Action scheduling, and execution-entity health metrics while preserving existing names and semantics.

4.1 Existing Baseline

Scope	Metric	Type	Meaning
Agent	numOfEventProcessed	Counter	Processed business Events
Agent	numOfEventProcessedPerSec	Meter	Business Event processing rate
Agent	numOfActionsExecuted	Counter	Completed Actions
Agent	numOfActionsExecutedPerSec	Meter	Action completion rate
Action	numOfActionsExecuted	Counter	Completed invocations of one Action
Action	numOfActionsExecutedPerSec	Meter	Completion rate of one Action
Agent	eventLogTruncatedEvents	Counter	Event Log records whose payload was truncated
Model	promptTokens	Counter	Prompt tokens consumed
Model	completionTokens	Counter	Completion tokens consumed
Model	totalTokens	Counter	Total tokens consumed

4.2 Input Run

An Input Run is one attempt to process an input for a business key. Metric tracking begins when the runtime receives the Input Event. The run's processing phase begins later, when that Event leaves the pending queue. Tracking ends when all associated Actions complete or an unhandled exception terminates the run.

Scope	Metric	Type	Meaning
Agent	numOfInputRunsSucceeded	Counter	Input Runs that reached the normal completion boundary
Agent	numOfInputRunsFailed	Counter	Input Runs terminated by an unhandled exception, including failures before run context creation
Agent	inputRunLatencyMs	Histogram	Input receipt to run completion or failure
Agent	inputRunQueueLatencyMs	Histogram	Input receipt to the start of run processing, primarily reflecting same-key queueing
Agent	inputRunProcessingLatencyMs	Histogram	Start of run processing to completion or failure
Agent	numOfPendingInputEvents	Gauge	Input Events waiting in same-key queues
Agent	numOfActiveInputRuns	Gauge	Runs selected for processing but not yet complete

The latency boundaries are:

Input Event received
    │
    ├─ inputRunQueueLatencyMs
    │
Input Run starts processing
    │
    ├─ inputRunProcessingLatencyMs
    │
Input Run completed or failed

inputRunLatencyMs = received → completed or failed

Here, processing starts when the Input Event leaves the pending queue and enters Event routing. It does not imply that the first Action has already begun executing.

A child execution failure does not by itself make the Input Run fail. If the Agent handles the failure and reaches its normal completion boundary, the run is successful.

4.3 Action

Action metrics distinguish between a physical ActionTask and a logical Action execution. One logical execution may suspend while waiting for asynchronous work and later resume through another ActionTask.

Scope	Metric	Type	Meaning
Action	actionSchedulingLatencyMs	Histogram	Initial ActionTask enqueue to its first execution
Action	actionExecutionLatencyMs	Histogram	Action execution started to finished or failed, including asynchronous waiting
Action	numOfPendingActionTasks	Gauge	Physical ActionTask instances waiting for execution, including continuations
Action	numOfActiveActionExecutions	Gauge	Logical Action executions that started but have not reached a terminal state

Only the initial ActionTask contributes a scheduling-latency sample. A continuation updates the pending-task Gauge but does not create another scheduling sample.

4.4 LLM and Token Usage

LLM health metrics use the configured ChatModel resource scope. Each sample represents one logical model call, not an individual provider attempt.

Scope	Metric	Type	Meaning
Model Resource	numOfLlmCallsSucceeded	Counter	Logical LLM calls that ultimately succeeded
Model Resource	numOfLlmCallsFailed	Counter	Logical LLM calls that ultimately failed
Model Resource	llmCallLatencyMs	Histogram	Logical call latency, including retry attempts and retry wait time
Model Resource	retryCount	Counter	Provider retries performed
Model Resource	retryWaitSec	Counter	Total retry wait time in seconds
Model	promptTokens	Counter	Prompt tokens attributed to the provider-returned model name
Model	completionTokens	Counter	Completion tokens attributed to the provider-returned model name
Model	totalTokens	Counter	Total tokens attributed to the provider-returned model name

Structured-output parsing is a separate execution in Agent Trace. If the model call succeeds but parsing fails, the LLM call remains successful. The Parser failure is diagnosed through Agent Trace rather than counted as an LLM failure.

4.5 Tool, Skill, and MCP Server

Each Tool call is one Tool execution. Skill and MCP Server metrics are additional aggregate projections derived from explicit Tool execution metadata; they do not introduce separate execution levels.

Scope	Metric	Type	Meaning
Tool	numOfToolCallsSucceeded	Counter	Successful Tool calls
Tool	numOfToolCallsFailed	Counter	Failed Tool calls
Tool	toolCallLatencyMs	Histogram	Tool-call latency
Skill	numOfSkillLoads	Counter	Explicit load_skill calls that reached a terminal state
Skill	skillLoadLatencyMs	Histogram	Explicit load_skill call latency
MCP Server	numOfMcpToolCallsSucceeded	Counter	Successful Tool calls exposed by one MCP Server
MCP Server	numOfMcpToolCallsFailed	Counter	Failed Tool calls exposed by one MCP Server
MCP Server	mcpToolCallLatencyMs	Histogram	Latency of Tool calls exposed by one MCP Server

Every named Tool execution contributes to Tool metrics. Only an explicit load_skill Tool execution contributes to Skill metrics. The runtime does not infer that subsequent Tool calls belong to the loaded Skill. A Tool execution carrying MCP Server metadata contributes to both its Tool and MCP Server scopes.

5. Recording Semantics

5.1 Event-Driven Metrics

Metrics derived from Business Events and Execution Events use the same semantic boundaries as Agent Trace:

• A processed business Event increments Event throughput metrics.
• Action Execution Events update Action execution latency and active-execution state.
• Terminal LLM and Tool Execution Events update outcome counters and latency.
• Tool execution metadata adds Skill and MCP Server aggregate projections where applicable.

execution_id may be used internally to pair started and terminal Events, but it is never exposed as a metric scope or label.

Metric aggregation consumes these in-process Events directly rather than reading them back from the Event Log.

5.2 Runtime State Metrics

Input Run and scheduling metrics require state transitions that are not fully represented by Execution Events:

• Input receipt, pending-queue enqueue, and pending-queue dequeue define run queue latency and pending counts.
• Run processing start and terminal boundaries define processing latency and active-run counts.
• ActionTask enqueue and dequeue define scheduling latency and pending-task counts.

These observations extend the metric model without turning queue operations into public Event types.

5.3 Model Call Metrics

Token usage and retry information are available only at the model-call boundary:

• Retry metrics record the actual number of retries and cumulative wait time for the logical call.
• Token metrics use usage data and the model name returned in the final response.

These values complement the LLM Execution Event without expanding its lifecycle schema with provider-specific payloads.

5.4 Outcome Boundaries

Scenario	Metric outcome
An LLM call succeeds after retries	One successful LLM call; latency includes retries and wait; retry counters increase
An LLM call succeeds but structured-output parsing fails	LLM success; Parser failure remains visible in Agent Trace
A Tool returns an error response	Failed Tool execution, even if the containing Action handles the response
A child execution fails but the Agent handles it	Child failure metrics increase; the Input Run can still succeed
An Action suspends and resumes through a continuation	One logical Action execution; continuation affects pending tasks but not scheduling sample count

6. Task Attempt and Recovery Semantics

Flink metrics are scoped to the current task attempt. Counter, Meter, and Histogram values are not restored from checkpoints and do not provide exactly-once values across failures.

Current-value Gauges may be reconstructed from restored runtime state when that state identifies pending or active work. Historical start timestamps cannot be reconstructed:

• A latency sample is recorded only when the current attempt observes both the required start and terminal boundaries.
• Restored pending or active work contributes to current-value Gauges without fabricating a historical start time.
• LLM and Tool terminal outcomes observed in the current attempt may update counters even when no latency sample can be produced.

Consumers must aggregate Gauge values across subtasks for a job-level view. Histograms include only samples observed by the current task attempts.

7. Cross-Language and Reporter Semantics

Java and Python use the same logical execution entity types, lifecycle outcomes, metadata keys, metric names, and scope hierarchy. This alignment is part of the metric contract, even though the two runtimes observe LLM and Tool calls at different language-specific boundaries.

MetricGroup scopes are the framework-level representation of dimensions. A reporter may expose them as labels or encode them in a metric identifier, but this does not change their logical meaning. Dashboards and queries must preserve or deliberately aggregate scopes rather than assume identical physical names across reporters.

8. Prototype Status

The metric model has been prototyped in both the Java and Python runtime paths. The prototype covers:

• Input Run outcome, latency, queue, and active-state metrics.
• Action scheduling, execution latency, pending-task, and active-execution metrics.
• Event-driven LLM and Tool outcome and latency metrics.
• Model Resource, model, Tool, Skill, and MCP Server scopes.
• Token usage and retry metrics.
• Task-attempt and recovery behavior for Gauges and latency samples.
• Java and Python semantic alignment through unit and runtime tests.

The purpose of this discussion is to review the metric contracts and their relationship to Agent Trace before submitting the implementation upstream.

9. Summary

Agent Trace and operational metrics answer different questions but share the same runtime semantics. Business Events and Execution Events provide stable observation boundaries for Event throughput and execution health. Run state, queue state, and model-call results provide the additional signals required by metrics.

This proposal adds:

• Input Run health and latency metrics.
• Action scheduling and execution-state metrics.
• LLM and Tool success, failure, and latency metrics.
• Model Resource, model, Tool, Skill, and MCP Server scopes.
• Explicit cardinality and task-attempt semantics.
• A shared metric contract for Java and Python.

The result is a bounded-cardinality operational metric surface that complements per-run Agent Trace without depending on Event Log storage or introducing per-run identifiers into metrics.