Exporting Events

Exporting Events

Operators already have a tracing tool, and an agent's runs are worth seeing in it beside everything else they watch. The runtime does not ship a dashboard. It ships two things that feed the one the operator has: export, a package that follows a database's events table from outside the process that owns it, and export/otel, a nested module that turns those events into OpenTelemetry spans for whatever collector the operator's OTEL_* environment names. The consumer writes no code for either. Both shipped in v0.3.1; export is in the core, and export/otel is its own module, tagged export/otel/v0.1.0, because it brings OpenTelemetry with it.

go get github.com/joeylking/agent-runtime/export/otel

The module's first tag, v0.1.0, required core v0.3.1, which is the first release with the export package its example imports. v0.1.1 requires core v0.4.0 and adds one span attribute, agentrt.decision.origin, which carries a decision's origin when its payload names one. v0.1.2 requires core v0.5.0 and adds two more, agentrt.policy.id and agentrt.policy.spec_hash.

This works because of a property the runtime already had: events are committed in the same transaction as the state change they explain, so the table is complete and a follower needs no other input. Nothing is reconstructed, and nothing needs the consumer's agent, tools, or policy.

The follower

export.Follower reads a store's events in commit order and hands each to a sink.

store, err := agentrt.OpenExisting("runs.db", true) // read-only, never creates, never migrates
f := &export.Follower{Store: store, Cursor: export.FileCursor{Path: "export.cursor"}}
err = f.Follow(ctx, export.JSONL(os.Stdout))

The cursor is the event's Seq. SQLite assigns it from an AUTOINCREMENT key inside the single write transaction that commits the event, so it rises in commit order across every run in the database and is never reused. A follower that remembers the last Seq it delivered resumes exactly after it.

It only reads. A follower opened on a store from OpenExisting(path, true) writes nothing and changes no file mode, so it can run as a separate process beside the consumer that owns the database, while that consumer is running. A read-only open can read a WAL database while a writer is live; see Persistence and Inspection.

It reads in bounded pages. Each query is Store.ListEventsAfter(ctx, runID, seq, limit), which returns at most limit events with a Seq greater than seq, for one run or, with an empty run id, for every run interleaved in commit order, with every text column capped at MaxPageText. A huge table, or one crafted row, cannot exhaust the follower's memory. The page is read and its query closed before any event is delivered, so the store's connection is never held across a sink call.

Field	Meaning
Store	required; read and never written
RunID	deliver one run's events only; empty delivers every run's
Cursor	where to resume from and save to; nil starts at the first event and remembers nothing beyond the call
Interval	the wait between polls once caught up: 250 ms by default (DefaultInterval), never more than a minute (MaxInterval)
PageSize	events per query: 500 by default (DefaultPageSize), never more than 10,000 (MaxPageSize)
Once	return nil as soon as everything committed has been delivered, instead of polling for more

A negative Interval or PageSize is an error. Follow returns ctx.Err() when its context ends it, and the sink's error when the sink fails.

Cursors. Cursor is two methods, Load and Save, so a consumer can keep the position wherever it keeps its own state. export.FileCursor holds the Seq in decimal in a file, written to a temporary file in the same directory, synced, given mode 0600, and renamed over the old one, so a crash leaves the previous cursor and never a torn file; a missing file loads as zero, and one that does not hold a number is an error. export.MemCursor keeps it in memory, for a follower that starts over each time or whose caller persists the Seq itself.

What delivery guarantees. The cursor is saved after each page and again before Follow returns, including when the context was cancelled or the sink failed: the save uses a context that outlives the cancellation, and it names the last event delivered, so a follower stopped by its context and started again does not deliver an event twice, and a restart after a sink error resumes with the event that failed. That is the exactly-once the changelog describes, and it holds across clean stops and restarts. A process killed between a sink call and the save that follows it delivers again from the saved cursor, so a sink that must not act twice keys on Seq.

The head of the chain

Since v0.5.0 the events are a hash chain (Re-checking a Run), and the follower can name the head of it as far as it has delivered:

seq, hash, err := f.Head(ctx)

Head returns the seq its cursor saved, which is the last event delivered, and that event's stored hash. It is zero and empty when nothing was delivered or the cursor is nil, and agentrt.ErrNotFound when the event the cursor names is no longer stored. It reads the cursor and one row and changes nothing, so it may be called while Follow runs, with a cursor safe for that, as FileCursor is. Events not yet delivered do not move it (TestFollower_HeadIsTheChainAsFarAsDelivered).

Keep the pair beside what the sink wrote, where whoever can write the database cannot reach, and check it later with agentrt verify -head SEQ:HASH (The Operator Command), which fails unless the chain reaches that seq with no break and the event there still has that hash. The hash covers every run's events up to that seq, a follower of one run included, because the chain is the whole database's. The JSON Lines record is unchanged: Event has no hash field (TestJSONL_RecordIsByteStable).

The JSON Lines sink

export.JSONL(w) is a Sink writing one JSON object per event to w: seq, run_id, step_id, at, type, and the payload as recorded. It is trace.JSONL behind a sink, so the two never differ, and a write error stops the follower. It is for a log shipper: follow a database from another process and pipe the lines wherever lines go. See Persistence and Inspection for the in-process observers.

The span mapping

export/otel's Exporter is fed by a follower: Exporter.Handle is an export.Sink. It maps a run's events onto spans, all of one run in one trace.

Span	Name	Starts at	Ends at
run	agentrt.run	run.created	run.finished
step	agentrt.step	step.started	step.tool_finished (done), step.failed, the next step.started of the run, or run.finished
model attempt	agentrt.model	model.dispatched	model.completed or model.failed
approval wait	agentrt.approval	approval.requested	approval.decided, or run.finished while still pending
late tool	agentrt.tool.late	a step.tool_finished marked late	the same event: what a tool returned to a cancelled run, as a child of its step
unknown event	agentrt.event	an event type the package does not know	the same event: a zero-length span under the run when the run span is not open; on the run span as an event when it is

Step spans are children of the run span, model spans of the step span, and approval spans of the step span. A span is emitted when it ends, so a live run's steps and model calls arrive as they complete and the run span arrives when the run finishes. step.interrupted does not end its step: an interrupted side effect may run again on the same step once an operator approves it, so the step ends with the rest of the run.

The span names are fixed strings, exported as SpanRun, SpanStep, SpanModel, SpanApproval, SpanLateTool, and SpanEvent, and nothing else ever names a span. The instrumentation scope of every span is github.com/joeylking/agent-runtime/export/otel (Scope).

Events that are not spans of their own are span events, named as the runtime names them, with their payload fields as attributes. On the run span: run.started, lease.taken_over, limit.exceeded, loop.detected, and run.resumed, and the run.finished of a cancelled run as run.cancelled. On the step span: step.decided, step.policy, step.tool_started, step.interrupted, and approval.requested. step.started is a span start and has no event.

Status. A span's status is Error for a FAILED run, a failed step, and a failed model attempt, Ok for a COMPLETED run, and unset otherwise.

Every event type the runtime emits is mapped; the exporter's tests run real drivers through a pause and a resume, an interruption and a takeover, a cancel with a late tool outcome, and a model-backed run with a retry, and check that each type lands somewhere (TestFixtures_EmitEveryEventType). Handle never returns an error: a payload that does not decode still ends the span it ends, with what could be read of it.

The attributes

Names are stable, and a release that changes one says so in the changelog. "text" marks a value cut at MaxText.

Attribute	On	From	Value
agentrt.run.id	every span	the event	run id
agentrt.step.id	step, model, approval, late	the event	step id
agentrt.run.goal	run	run.created	text
agentrt.limits.max_steps	run	run.created	int
agentrt.limits.max_consecutive_tool_failures	run	run.created	int
agentrt.limits.loop_threshold	run	run.created	int
agentrt.limits.max_model_calls	run	run.created	int, when set
agentrt.limits.max_output_tokens_per_call	run	run.created	int, when set
agentrt.limits.max_total_tokens	run	run.created	int, when set
agentrt.limits.max_estimated_cost_micros	run	run.created	int, when set
agentrt.limits.max_active_time_ms	run	run.created	int, when set
agentrt.limits.max_elapsed_time_ms	run	run.created	int, when set
agentrt.limits.approval_ttl_ms	run	run.created	int, when set
agentrt.limits.grant_ttl_ms	run	run.created	int, when set
agentrt.run.status	run	run.finished	COMPLETED, FAILED, CANCELLED
agentrt.run.reason	run	run.finished	terminal reason
agentrt.run.detail	run	run.finished	text
agentrt.run.steps	run	run.finished	int
agentrt.run.by	run	run.finished	who cancelled, when an operator did
agentrt.step.index	step	step.started	int
agentrt.decision.kind	step	step.decided	tool_call, complete, fail, truncated, no_tool_call
agentrt.decision.tool	step	step.decided	tool name as the agent gave it (text)
agentrt.decision.args	step	step.decided	JSON, text
agentrt.decision.reason	step	step.decided	text
agentrt.decision.result	step	step.decided	JSON, text
agentrt.decision.message	step	step.decided	text
agentrt.decision.invalid_args	step	step.decided	text, when the arguments were not JSON
agentrt.decision.invalid_result	step	step.decided	text, when the result was not JSON
agentrt.decision.invalid_base64	step	step.decided	bool, when set
agentrt.policy.outcome	step	step.policy	allow, deny, abort, require_approval
agentrt.policy.reason	step	step.policy	text
agentrt.policy.kind	step	step.policy	approval kind, for require_approval
agentrt.policy.id	step	step.policy	the policy's identity, when the policy recorded one
agentrt.policy.spec_hash	step	step.policy	SHA-256 of the tool spec the policy was handed
agentrt.tool.name	step, late	step.tool_started, step.tool_finished	registered tool name
agentrt.tool.duration_ms	step, late	step.tool_finished	int
agentrt.tool.summary	step, late	step.tool_finished	text
agentrt.tool.content_hash	step, late	step.tool_finished	SHA-256 of the observation
agentrt.tool.error	step	step.tool_finished	text
agentrt.tool.observation	late	step.tool_finished	JSON, text
agentrt.step.status	step	the end event	done, failed, or interrupted
agentrt.step.end	step	the end event	its type
agentrt.step.detail	step	step.failed	text
agentrt.step.observation	step	step.failed	observation kind
agentrt.step.content_hash	step	step.failed	SHA-256 of the observation
agentrt.step.previous_status	step event	step.interrupted	deciding, awaiting_approval, executing
agentrt.approval.id	approval, step event	approval.*	approval id
agentrt.approval.kind	approval	approval.requested	kind
agentrt.approval.reason	approval	approval.requested	text
agentrt.approval.hash	approval	approval.requested	the binding hash
agentrt.approval.capability	approval	approval.requested	JSON, text
agentrt.approval.presentation	approval	approval.requested	JSON, text
agentrt.approval.status	approval	approval.decided	approved, rejected, expired; pending at run end
agentrt.approval.by	approval	approval.decided	who decided
agentrt.approval.note	approval	approval.decided	text
agentrt.model.call_id	model	model.dispatched	attempt id
agentrt.model.attempt	model	model.dispatched	int, 1-based within the step
agentrt.model.name	model	model.dispatched	model name
agentrt.model.status	model	model.completed, model.failed	ok, error, ambiguous
agentrt.model.input_tokens	model	model.completed, model.failed	int
agentrt.model.output_tokens	model	model.completed, model.failed	int
agentrt.model.cached_input_tokens	model	model.completed, model.failed	int
agentrt.model.cost_micros	model	model.completed, model.failed	int
agentrt.model.latency_ms	model	model.completed, model.failed	int
agentrt.model.error	model	model.failed	text
agentrt.lease.previous_owner	run event	lease.taken_over	owner
agentrt.lease.previous_expires_at	run event	lease.taken_over	RFC 3339
agentrt.lease.owner	run event	lease.taken_over	owner
agentrt.limit.reason	run event	limit.exceeded	terminal reason
agentrt.limit.detail	run event	limit.exceeded	text
agentrt.loop.repeats	run event	loop.detected	int
agentrt.loop.tool	run event	loop.detected	tool name
agentrt.loop.args_hash	run event	loop.detected	SHA-256
agentrt.loop.observation_hash	run event	loop.detected	SHA-256
agentrt.resumed.after	run event	run.resumed	"interruption", when resumed after one
agentrt.event.type	event	an unknown event	its type
agentrt.event.seq	event	an unknown event	its Seq
agentrt.span.start_unknown	any	a missing start	true when neither the event nor Store gave the start
agentrt.payload.truncated	any	any	full length of a payload the follower capped

Deterministic ids

Every span of a run is in one trace whose id is derived from the run's id, and every span id is derived from the id of what it stands for, so an exporter restarted, or a run resumed days later, continues the trace instead of forking it. otel.TraceID(runID) is the first 16 bytes of the SHA-256 of "agentrt/trace", a zero byte, and the run id. otel.SpanID(kind, id) is the first 8 bytes of the SHA-256 of "agentrt/span", a zero byte, the kind (run, step, model, approval, or late), a zero byte, and the id of the run, step, attempt, approval, or late step. An all-zero result is nudged to a valid id.

Two exporter processes, one sending over http and one over gRPC, produced byte-identical trace and span ids (the proof run below).

A provider has to use the generator. otel.IDGenerator() is the SDK id generator that honours these derivations and otherwise draws random ids as the SDK's own does. otel.Provider and otel.NewTracerProvider install it; a TracerProvider built any other way must install it itself, or the spans get random ids and a restart forks the trace. An Exporter with a nil TracerProvider traces on the global provider, which then needs the generator.

Restarts

An exporter that did not see a span's start, because it was started after the event, takes the start from its Store when one is given: the run's creation time, the step's start time as stored (which, for a step resumed after an approval, is when it resumed), the approval's creation time. A model attempt's start is its end less its latency, and a step ended by step.tool_finished starts its duration before it, store or not. Each such lookup is bounded at five seconds. Without a start from either, the span starts at the first of its events this exporter saw and carries agentrt.span.start_unknown.

The Store is only read, and the follower's read-only store serves.

The Exporter

exporter := &otel.Exporter{TracerProvider: tp, Store: store}
err := (&export.Follower{Store: store, Cursor: cursor}).Follow(ctx, exporter.Handle)

Field	Meaning
TracerProvider	produces the spans; nil is the global provider
Store	read for the start of a span whose start event this exporter did not see
MaxText	the bound on model-chosen text in an attribute; zero is 1024 bytes (DefaultMaxText)

The zero value is usable. One Exporter must see a run's events in order, as a follower delivers them, and is safe for concurrent use.

The OTEL environment

otel.Provider(ctx) builds a TracerProvider from the standard environment, with the deterministic id generator installed and spans batched. Shut it down to flush what is batched.

Variable	Effect
OTEL_TRACES_EXPORTER	otlp (the default), console, or none; anything else is an error
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, then OTEL_EXPORTER_OTLP_PROTOCOL	http/protobuf (the default) or grpc; http/json is not built in
OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, and the rest of the OTLP exporter's own variables (headers, timeout, insecure, certificate)	read by the OTLP exporter; the defaults are a stock collector's http://localhost:4318/v1/traces and localhost:4317, where gRPC uses TLS unless the endpoint's scheme is http or the insecure setting is true
OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES	read into the resource; the default service.name is agentrt

Options passed to Provider are applied after the exporter and the resource, so they may replace either.

Running the example beside a consumer

export/otel/example is the whole operator-side program: it opens the database read-only, follows the events table, and sends spans to the collector the environment names. The consumer needs no code and need not stop.

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 go run ./export/otel/example -db runs.db

Flag	Meaning
-db	the SQLite file; defaults to $AGENTRT_DB
-run	export one run only; the default is every run
-cursor	a file holding the follower's position, so the next start resumes after the last event exported; without it the follower starts from the first event and remembers nothing
-once	export what is committed and exit
-interval	the wait between polls when caught up
-v	print each event's type to stderr

Ctrl-C stops it after flushing the provider, with a ten-second deadline of its own, because the context that stopped it is already cancelled. The example is a nested module, so running it from a clone needs the workspace; see Getting Started.

The cursor advances when events have been handed to the exporter, and the provider batches what it sends. The example flushes on a clean stop; the repository states no stronger guarantee for spans a killed exporter had batched and not yet sent.

What is truncated, and what still leaves the machine

Nothing a model chose ever names a span: the goal, a reason, a summary, arguments, a message, and a provider's error text are carried in attributes, cut at MaxText bytes on a rune boundary with a marker giving the full length (otel.Truncate is the cut). Everything else is carried whole, except that the follower caps a payload at MaxPageText, which the span then records as agentrt.payload.truncated.

That is a bound on size and on names, not redaction. What remains, decision arguments, tool summaries, an approval's capability and presentation, and a provider's error text, is still sent to whatever collector the operator configured. The runtime does not redact it, and a collector is outside what the runtime defends. A tool that returns a secret has already put it in the database; the exporter forwards a bounded copy of the parts it maps. See Security Model.

What was checked

On 2026-10-02 the example's output was sent to otel/opentelemetry-collector-contrib v0.161.0 in a container, and a run's trace appeared there with no code in the consumer, which is the roadmap's done-when for event export. The module's direct dependencies are the core, go.opentelemetry.io/otel, otel/sdk, otel/trace, otlptracehttp, otlptracegrpc, and stdouttrace; its module graph is 114 modules, 35 of which provide packages. That is the cost the nested module exists to keep out of the core.

Next

Approval Channels is the other thing an operator surface does with a database it does not own. Persistence and Inspection covers the table being followed, and Security Model what leaves the machine.