Runtime Observability

Runtime Observability

AISMixer exposes pull-based runtime statistics for the current service process. The snapshots help operators interpret queue pressure, processor activity, local egress work, and transport traffic without changing runtime state.

These statistics are:

• fresh immutable snapshots;
• process-local and held only in memory;
• current gauges plus lifetime counters for the existing component instances;
• non-durable and restarted from zero with a new service process; and
• observational: reading them does not reset counters or mutate routing or processing state.

The aggregate provider reads its owners once in a fixed sequence. It does not stop the runtime while doing so, so one aggregate result is not a transactionally atomic observation of every independently changing component. No persistence, time-series store, push exporter, rate calculation, cross-process aggregation, or distributed monitoring system is implied.

Queue Statistics

The aggregate snapshot includes every private ingress queue, the shared processing-admission queue, and the egress queue. Each queue reports:

Field	Interpretation
capacity	Maximum number of items, not bytes.
depth	Items currently queued. Active or reader-held work is not necessarily included.
peak_depth	Highest depth in the current queue instance's lifetime.
enqueued / dequeued	Successful queue or admission operations.
put_waits	Historical put or admission attempts that initially found no capacity. Cancelled waits remain historical.
current_put_waiters	Such waits still outstanding when the snapshot was read.

Queue depth alone is not the total amount of in-flight work. In particular, a fan-in reader may hold one accepted frame while waiting for processing capacity, and the processor may be active after its work item has left the processing queue. See Architecture Overview for the bounded queue topology and backpressure semantics.

Processor Statistics

The one long-lived PythonDataPlaneProcessor reports:

• process calls that are completed, failed, or currently in flight;
• successful calls with no output;
• non-empty output batches and their total ProcessorOutput count; and
• reset calls that are completed, failed, or currently in flight.

Processor output counters describe results constructed synchronously by the processor. They are not counts of destination dispatches or remote deliveries. Processor reset retains these cumulative metrics.

Egress-Operation Statistics

The egress stage reports batch and output operations as started, completed, failed, cancelled, or currently active. One output operation corresponds to one ProcessorOutput, regardless of how many numeric targets that output contains. A processor call with no output creates no egress operation.

Completion means the awaited local egress operation returned. It does not mean that a remote UDP consumer received, accepted, or processed a datagram.

Input Traffic Statistics

Each configured UDP or UDPSEC input has a row containing its runtime name and kind. Two pairs of counters deliberately describe different boundaries:

transport_packets / transport_bytes
    raw datagrams observed immediately after socket receive

accepted_frames / payload_bytes
    constructed frames counted only after private ingress-queue admission

Raw transport traffic can therefore exceed accepted frame traffic. Depending on the input kind, it can include denied, malformed, handshake, or other traffic that never becomes an admitted IngressFrame. This distinction is useful when diagnosing noisy or unauthenticated input without mistaking every received datagram for processor work.

Output Traffic Statistics

Every configured egress destination has one row in numeric target order, including unnamed legacy destinations. A row contains the process-local numeric target_id, its optional external name, and:

• dispatch_attempts;
• dispatch_completed;
• dispatch_failed;
• successfully dispatched messages; and
• successfully dispatched payload bytes.

An attempt is counted for each selected target before local transport setup or send. Only a successful local return increments completion, messages, and bytes. Again, dispatch_completed is not acknowledgement of remote UDP delivery.

Reading Statistics

The optional local control protocol exposes three read-only methods:

Method	View
runtime.statistics	Ingress queues, processing queue, processor, egress queue, and egress operations.
runtime.statistics.inputs	Detailed input traffic, optionally filtered by exact input name.
runtime.statistics.outputs	Detailed output traffic, optionally filtered by exact numeric target ID or external name.

A detailed filter with no match returns an empty list. These reads neither reset statistics nor affect routing. Conversely, routing status, replacement, and disable operations neither pull nor reset statistics.

aismixerctl exposes the same views as show statistics, show statistics inputs [INPUT], and show statistics outputs [OUTPUT]. See Runtime Routing Control for protocol parameters, one-shot use, and the interactive shell.

Exact field invariants and edge-case accounting remain normative in the main repository behavioural contract.