Agent Auditing

[igordayen]

To explain agent behavior properly, it's usually required to track:

a. action execution chain
b. blackboard state
c. LLM Invocations
d. tool Invocations
e. external events

An observability framework provides a solid foundation for tracking.
Blackboard state tracking on a granular level per object requires some additional effort.

Auditing / History tracking would also allow for action replay.

An auditing framework can serve as a low-level foundation for Proposition revisions management.
An auditing framework can serve as a foundation for Evolving Mode, to adjust conditions/heuristics based on history.

Additional thoughts:
a. capture agent audit in file system for further analysis/replay
b. long-running actions - is it possible scenario? - to be suspended/restarted from a savepoint?

@johnsonr , @alexheifetz @azanux @Stuckya @jimador @jasperblues - FYI

[Stuckya]

Looks great.

A suggestion from our evolving mode conversations. Make the unit of audit a single attempt. For today that's just the process run. In a long-running process, there are many attempts (what we have been calling 'episodes').

Regarding 'to be suspended/restarted from a savepoint', I don't think we need this. Since we've been discussing breaking long running work into episodes, they will write their progress to the blackboard. The audit tells us why the last attempt died. The blackboard tells the next attempt where to start. Seems easier than making actions resumable.

+1 to filesystem capture as the first durability experiment.

[igordayen]

thanks @Stuckya
Make the unit of audit a single attempt. For today, that's just the process run. In a long-running process, there are many attempts (what we have been calling 'episodes').
==> Yes, per occurrence.

[jimador]

This is great. I am going to try to capture some formatted markdown with the problem statement, the existing landscape within the agent framework, the gaps that we currently have that I can quickly identify, and what some of the external landscape looks like.

+1 to the episode as the unit of audit, the blackboard carrying continuation state instead of resumable actions, and filesystem capture as the first durability experiment.

Problem statement

Two consumers for the same record:

1. Explain the past. Why did this run produce this result, why did the planner pick this chain, what did action X see and change, why did the last attempt die.
2. Improve the future. Evolving mode. Adjust action costs, conditions, heuristics, and prompts from accumulated history.

The second depends on the first. Every improvement loop in the literature consumes the records an audit trail provides (table further down), so the record model should cover both uses from the start.

What we already have

The event layer covers most of the capture list:

Audit need	Existing signal
a. Action execution chain	ActionExecutionStartEvent / ActionExecutionResultEvent, plus AgentProcess.history
b. Blackboard state	ObjectAddedEvent / ObjectBoundEvent per write, WorldState on the planning events (incomplete, see gaps)
c. LLM invocations	LlmRequestEvent / LlmResponseEvent with full message lists, ChatModelCallEvent with the final augmented prompt, LlmInvocation with usage and cost
d. Tool invocations	ToolCallRequestEvent / ToolCallResponseEvent, tool loop start/complete events
e. External events	nothing first-class today
Planning decisions	AgentProcessReadyToPlanEvent, AgentProcessPlanFormulatedEvent (world state plus chosen plan), ReplanRequestedEvent

The hook point also already exists. AgenticEventListener is auto-aggregated from Spring beans and multicast per process, and the observability module (EmbabelSpanEventListener, EmbabelMetricsEventListener) already consumes every event and projects it into spans and metrics. An audit recorder can follow the same pattern and write to an append-only journal instead. Since embabel-agent-observability already emits OTel-style spans, audit records should be convertible to OTel spans rather than replacing that layer.

Gaps I can quickly identify

1. No durable sink. InMemoryAgentProcessRepository is a bounded, eviction-windowed cache (nothing survives the JVM).
2. Snapshot serialization is blocked by design. AgentProcess is annotated with ComputerSaysNoSerializer. I think that's the right call. We should event-source the audit trail and never serialize the live process graph.
3. The event stream is not a complete blackboard journal. Initial bindings are applied at process creation before any listener can observe them, clear() and hide() emit nothing, and in-place mutation of an already-bound object is invisible. Reconstructing blackboard state from today's binding events alone is not possible.
4. Events are views over the live process. AbstractAgentProcessEvent computes history and status from the process at read time, and binding events hold live object references. A recorder has to materialize an immutable event-time envelope synchronously before any async buffering.
5. No causal identity under concurrency. ConcurrentAgentProcess runs actions concurrently and binding events carry only the process id, so interleaved writes cannot be attributed to an action by timestamps. Action retries happen inside a single start/result pair with no per-attempt event. Per-occurrence granularity needs a step/attempt id threaded onto the events themselves.
6. Episode identity. A process id is not an attempt. A process can pass through waiting/paused/stuck and run again, so close/reopen semantics need a decision, and sibling attempts at the same goal have no linkage at all.
7. AgentProcessFailedEvent is an empty subclass. Failure detail lives only on the live process, so "why did the last attempt die" is not answerable from the event stream today.
8. Plan alternatives are discarded. plansToGoals() ranks all goal plans and bestValuePlanToAnyGoal() throws the ranking away. Answering "why this plan" later means keeping the alternatives, which looks to be a planner API change (@johnsonr confirm?).
9. No statistics surface. CostComputation is already a function of WorldState, which is a good substitution point for learned costs, but nothing accumulates per-action success, latency, or cost across runs.
10. Trigger provenance. Root process creation takes bindings and options with no cause field, so "what caused this episode" (item e) needs a small creation-envelope change. The broader event-to-trigger subscription surface is worth its own discussion.

One more, operational: MulticastAgenticEventListener swallows listener failures. Right for metrics, but leaves silent gaps for an audit trail. The recorder needs an explicit best-effort vs mandatory mode, with gap records when capture degrades.

External landscape

• The OTel GenAI semantic conventions define invoke_agent / execute_tool / generation spans (still development status), and Spring AI implements them via Micrometer, splitting low-cardinality attributes from high-cardinality payloads that are off by default. This could be a good template for blackboard payload policy. See https://opentelemetry.io/docs/specs/semconv/gen-ai/ and https://docs.spring.io/spring-ai/reference/observability/index.html
• The tracing/eval platforms (LangSmith, Langfuse, Phoenix, Weave, Braintrust) seem to be converging on the same data model: a session/trace of nested typed spans, with evaluations attached as scored annotations keyed to a span id, mostly post-hoc. Per-step stable ids look to be the eval attachment point everywhere.
• Replay systems (LangGraph checkpointing/time-travel, Temporal, Restate) are event sourcing plus some determinism discipline. Explain-replay is a reader over the journal. Re-execution replay additionally needs recorded results substituted for LLM/tool calls and an idempotency story for side effects, so it is a separate contract for a later phase.
• None of these systems capture per-action working-memory diffs. Blackboard writes already go through a single code path here, so we could support this cheaply once gap 3 is closed.

What evolving mode needs recorded

Reduced from the self-improvement literature (Reflexion arXiv:2303.11366, ExpeL arXiv:2308.10144, trace-based prompt optimization like GEPA arXiv:2507.19457, trajectory evals, classical action-model learning from execution failures arXiv:2103.05682):

Without	You cannot
Per-step pre/post state	Learn or adjust preconditions and effects from failures
Per-action cost/latency/tokens	Learn action costs or heuristics
Outcome labels per action and per episode	Reflection or insight extraction
Full LLM/tool payloads	Trace-based prompt optimization
Stable action identity, including model/prompt/tool versions	Any trustworthy cross-run statistic
Episode boundaries	Episodic memory, anything beyond RAG over transcripts

There are two distinct feedback targets. Numeric planner learning (success rates, observed costs) is deterministic aggregation, needs no payload capture, and fits GOAP well since costs and conditions are already part of the action model. Knowledge-level learning (heuristics distilled from episodes by eval passes) is a text artifact that needs provenance, confidence, and a lifecycle.

Proposed shape

Phases, riskiest first:

0. Event envelope: immutable event-time capture, causal step/attempt ids, a schema version, the missing emissions (initial bindings, clear/hide, failure detail on the failed event, the streaming LLM path), and explicit loss semantics.
1. Episode assembly, filesystem JSONL journal, payload policy (structure always on, payloads per-type opt-in with redaction).
2. Explain-replay and eval attachment keyed to step ids.
3. Statistics/query surface over accumulated episodes, per-action marginals first.
4. Feedback: statistics-informed costs behind a flag, and a heuristic store behind the sink SPI.

Non-goals: process-graph serialization, resumable actions (episodes plus blackboard, as agreed above), re-execution replay in v1, replacing the OTel observability layer.

DICE as one feedback-store implementation

dice is a good fit for phase 4, and the coupling works out cleanly because dice already depends on embabel-agent. The audit SPI (record model, sink, listener) lives in embabel-agent, dice or a small integration module implements the sink/feedback side, and the framework never references dice. Any other store can implement the same SPI.

What a proposition store adds to the evolving loop: heuristics stored as propositions get confidence, decay, reinforcement, contradiction handling, and provenance back to the episodes that produced them. One caveat from reading the dice model with this use in mind - reinforcement today counts evidence merges and retrieval, and decay is age/access based, so a frequently retrieved but harmful heuristic would stay fresh. For heuristics, reinforcement needs to be linked to episode outcomes (reinforce when injected and the episode succeeded, contradict when implicated in a failure... which is what we should get from the audit trail).

There is some working prior art from me here on the governance side in arc-mem which is a governed working-memory layer built on dice and embabel-agent. Promoted propositions carry an activation score, a trust/authority tier (top tier human-only), and membership in a hard-bounded active set injected into every prompt, behind a gated promotion pipeline (confidence, dedup, conflict, trust, budget). It governs conversational working memory rather than planner heuristics, so generalizing it is still an open question... but at least two lessons transfer here... 1., the promotion pipeline is a concrete answer to how evolving mode accepts learned heuristics without accreting garbage. And B., its simulator results (~700 adversarial runs) showed most of the benefit came from just having the bounded-governed-set in the context at all, with the individual policies mattering less (a note from the author here -- I feel like this isn't the entire story, but I haven't had enough time to go back and revisit each policy independently with any sort of tests that I would rely on with any sort of statistical significance to make me make any other claim). That said, I think evolving mode should optimize for a small distilled heuristic set with the audit journal as evidence, not for replaying raw history into context.

---
title: Evolving mode feedback loop
---
flowchart LR
    EP[episode journal] --> AGG[numeric aggregation]
    EP --> REF[eval passes / distillation]
    AGG -.-> COST[learned action costs]
    REF --> HP[heuristic store with provenance]
    HP --> GOV[governed promotion, bounded set]
    GOV --> RUN[next episode]
    COST -.-> RUN
    RUN --> EP
    RUN -- outcome-linked reinforce/contradict --> HP

Loading

[igordayen]

@jimador - great write up, James.
Linking discussions:

embabel/dice#75

Re: "audit records should be convertible to OTel spans rather than replacing that layer." - i thought to hook into observability events and create Audit Event from it.
Whatever is not covered - such as Blackboard state initiation and modification - develop it separately.

Will come with inquiries upon detailed analysis.

Thank you

[Stuckya]

Great write up James.

Gap 6 is the one I've been living in. Agreed, a process id is not an attempt. The close/reopen decision I've landed on is that attempts never reopen. An attempt runs from entry state to terminal state, gets its own id, and ends with exactly one outcome. A process that goes stuck and runs again has produced two attempts. Sibling attempts link through the piece of work they serve, not the goal (the same goal can serve many jobs). So the work needs a durable id too. One job, many attempts. I have a working POC branch that brings this split to life.

Gap 7 is the same item from my list and also in Igor's earlier thread #961 . Today a STUCK record doesn't say why. I think the planner should report its own obstruction rather than anyone inferring it from the trace.

On the two feedback targets, agreed. History should adjust the next attempt and shouldn't mutate an already running attempt. CostComputation as the substitution point seems to respect that nicely.

Your DICE caveat is where attempt ids pay off for you. Outcome-linked reinforcement needs each heuristic to point at the attempts that produced it, then reinforce or contradict their terminal outcomes. That join only exists if attempts have stable ids and outcomes.

+1 to evolving mode distilling a small set of heuristics.