feat(observability): enrich execution facts for downstream reasoner-level analysis

[santoshkumarradha]

Summary

Strengthen the execution observability contract so downstream systems can do reasoner-level trend analysis without AgentField shipping a built-in regression engine.

This is not about adding slope math or alert policy to core. It is about making terminal execution facts clean, stable, and available on the right rails.

Why

Today AF is pretty good at per-execution visibility, but not yet strong enough at exposing a stable fact shape for downstream behavioral analysis over time.

The missing piece is not "analyzer logic in core". The missing piece is a contract that makes it easy for users to:

• consume terminal execution facts in real time via webhook
• backfill or bootstrap historical analysis via execution history/export
• group by reasoner_id as the main identity
• optionally segment further using caller-provided execution context

This keeps AF focused on facts, not interpretation.

Scope

Define and implement a stable execution fact shape for downstream analysis, centered on:

• execution_id
• workflow_id / run_id
• agent_node_id
• reasoner_id
• status
• status_reason
• started_at
• completed_at
• duration_ms
• retry_count
• error_category
• session_id
• actor_id
• execution context as caller-provided grouping metadata

Proposed rails

1. Observability webhook

Enrich terminal execution events so downstream consumers can analyze completed runs directly from the webhook stream.

Target event types:

• execution.completed
• execution.failed
• execution.cancelled

Optional:

• execution.started if useful for consumers that want early visibility before terminal state

2. Execution history / export

Expose the same core fact shape through a stable read path so consumers can backfill or bootstrap historical analysis without replaying webhook logs.

Exact endpoint shape can be decided in implementation, but it should support:

• filtering by reasoner_id
• time-bounded reads
• cursor pagination

Important design notes

• reasoner_id is the primary identity for this work
• AF should not introduce a platform-owned task_class concept here
• if callers want finer segmentation inside a reasoner, that should ride in execution context
• this issue does not add slope computation, thresholds, or alert semantics to core
• Prometheus additions can be evaluated separately; they are adjacent, not the main contract work

Example

The same reasoner_id might be invoked with different caller-defined analysis groups:

{
  "input": { "...": "..." },
  "context": {
    "analysis_group": "summary.short_form"
  }
}

{
  "input": { "...": "..." },
  "context": {
    "analysis_group": "summary.long_form"
  }
}

Downstream consumers can then analyze:

• all executions for reasoner_id = summarizer
• or only executions where context.analysis_group = summary.short_form

without AF needing to hardcode a universal taxonomy.

Acceptance criteria

• terminal execution webhook events include the agreed execution fact shape
• execution history/export exposes the same core fields with a stable contract
• reasoner_id is consistently available as the main downstream analysis key
• retry_count and error_category are available on terminal execution facts
• execution context is available for caller-defined grouping metadata
• docs explain webhook-first real-time analysis and history/export backfill usage
• no built-in regression analyzer, slope computation, or alert policy is introduced in core as part of this issue

Follow-up

Once this lands, issue #329 can close as resolved by primitives:

• downstream trend analysis becomes easy to build
• richer analytics and alerting remain outside core

[santoshkumarradha]

CC @AbirAbbas

[nanookclaw]

Clean decomposition. The four deliverables map directly to what a downstream trend analyzer needs:

• reasoner_id + config_hash → grouping key for apples-to-apples comparison
• context propagation → lets the caller tag executions without AF needing to know the taxonomy
• Versioned schema → downstream can pin to a fact shape and not break on upgrades

One thought on config_hash: if it covers the full reasoner config (model, temperature, tool set), it doubles as a "was this a fair comparison?" check. A trend line that mixes gpt-4o and claude-sonnet runs is noise, not signal. The hash makes that separation automatic.

Happy to help spec the schema contract or write the first downstream consumer against it once the fact shape stabilizes.