Skip to content

event envelope

Jump to bottom
Kadyapam edited this page Jun 4, 2026 · 7 revisions

Event envelope

The wire format for POST /api/events — what workers (and CLI in distributed mode) send to ingest events into the event log.

Current shape (noetl-server v2.9.0+, after the noetl-events crate adoption)

pub struct EventRequest {
    pub execution_id: String,        // i64 on the wire is a String (browser JSON-number precision)
    pub step: String,
    #[serde(alias = "name")]
    pub event_type: String,
    #[serde(default, alias = "context")]
    pub payload: serde_json::Value,
    pub meta: Option<serde_json::Value>,
    pub worker_id: Option<String>,
    pub result_kind: String,         // "data" | "ref" | "refs"  (server-only)
    pub result_uri: Option<String>,  // server-only
    pub event_ids: Option<Vec<i64>>, // server-only
    pub actionable: bool,            // server-only — orchestrator dispatch gate
    pub informative: bool,           // server-only — log-only persistence gate
    pub event_id: Option<String>,    // app-side snowflake per observability.md Principle 3
    pub status: Option<String>,      // STARTED / RUNNING / COMPLETED / FAILED
    pub created_at: Option<DateTime<Utc>>,
}

Since noetl-server v2.9.0 (PR noetl/server#38), the SHARED SUBSET of EventRequest is anchored to the canonical noetl_events::ExecutorEvent envelope via direct noetl-events = "0.1" dep + bidirectional conversion impls. See the noetl-events crate adoption section below for the design call and the four wire-compat tests that guard the boundary.

Three layered serde aliases keep pre-EE clients working without changes:

  • name → event_type (legacy field name)
  • context → payload (executor producers send context; server stores as payload)
  • Pre-EE clients that omit event_id / status / created_at fall back to server-side defaults (DB snowflake / name-derivation / Utc::now()).

Mirrors the Python EventEmitRequest shape — both servers accept the same wire format.

Cross-stack reconciliation — progress

Tracked on noetl/ai-meta#30 — Appendix H Rust migration umbrella. Reconciliation aligns four shapes (worker WorkerEvent, Python EventEmitRequest, Rust EventRequest, executor ExecutorEvent) so all accept the same wire format and worker switching is a one-liner.

PR Repo Status
EE-1 noetl/cli ✅ merged (#37) — executor 0.3.1 enriches ExecutorEvent with optional event_id / worker_id / meta + payload serde alias
EE-2 noetl/server ✅ merged (#6) — this server's EventRequest rename + new optional fields (noetl-server 2.0.0; pipeline-fix #7 published 2.0.1)
EE-4 noetl/noetl (Python) ✅ merged (#639) — Python EventEmitRequest aliases + worker_id top-level field + EventType: Literal[...] → str (the Literal was already out of sync with the dot-notation event types used throughout production Python code)
EE-3 noetl/worker ✅ merged (#11) — noetl-worker 3.0.0: WorkerEvent replaced with ExecutorEvent re-export from noetl_executor::events; EventEmitter stamps worker_id + created_at at the source per observability.md Principle 4

✅ Series complete + validated end-to-end (kind-noetl, 2026-05-31). All four producers / consumers now emit + accept the same wire format on /api/events.

EE-4 finalisation post-merge

EE-4 (noetl/noetl#639) added EventEmitRequest with validation_alias=AliasChoices(...) declarations but the schema lived in dead code — the actually-mounted /api/events endpoint at core/events.py::handle_event used the legacy EventRequest model with no aliases. Surfaced during the noetl-worker (Rust) kind-validation pass: the EE-3 wire shape (event_type, context, execution_id: i64) failed Pydantic validation at the live endpoint.

noetl/noetl#641 (noetl 4.0.0) closes the gap by adding the same validation_alias declarations to core/models.py::EventRequest:

  • name ← event_type
  • step ← node_name
  • payload ← context
  • execution_id accepts JSON integer OR string (mode='before' coercion)

After this fix the EE umbrella's wire-shape promise truly holds — every producer's events land cleanly in noetl.event with the EE-3 fields populated. Kind validation summary on noetl/ai-meta#30.

noetl-events crate adoption (EE-4 follow-up, noetl/ai-meta#49)

After the historical EE-1..EE-4 series above landed, the wire-format envelope still lived in two places: noetl-executor::events::ExecutorEvent (the producer-side source) and noetl-server's hand-aligned EventRequest (the consumer-side mirror, kept in sync by code review). When the Rust server started producing events itself (Phase D R2 orchestrator wiring, noetl/server#31), the duplicated shape became actively maintenance-heavy.

The follow-up tracked under noetl/ai-meta#49 extracted the envelope into a dedicated noetl-events crate:

Round PR What
1 noetl/cli#49 Carve ExecutorEvent + EventSink + EventEmitter + NoopSink out of noetl-executor::events into the dedicated noetl-events workspace crate. noetl-executor::events becomes a 1-line re-export so existing call sites compile unchanged.
2 noetl/cli#50 Crates.io publish prep + actual publish: noetl-events 0.1.0 first release, noetl-executor bumped to 0.4.0.
3 noetl/server#38 This server takes a direct noetl-events = "0.1" dep + adds From<ExecutorEvent> for EventRequest + TryFrom<&EventRequest> for ExecutorEvent impls so the shared subset is anchored to the canonical envelope. Server bumps to v2.9.0.

Why the two types stay distinct (not a literal struct swap)

EventRequest carries a strictly larger field set than the canonical envelope. Five of those fields are legitimately server-only:

Field Why server-only
result_kind Drives the constraint-compliant {status, reference} / {status, context} result shape (noetl/server#29).
result_uri Same: required when result_kind is "ref".
event_ids Same: required when result_kind is "refs".
actionable Controls whether the orchestrator dispatches commands on this event.
informative Controls whether the event is log-only.

Wire format also differs in two places: EventRequest encodes execution_id and event_id as String (browser JSON-number precision concession for the dashboard); ExecutorEvent uses i64 for both.

So PR 3 didn't replace EventRequest with ExecutorEvent. Instead, the shared subset (execution_id, step, event_type, payload/context, meta, worker_id, event_id, status, created_at) is now anchored to noetl_events::ExecutorEvent via the conversion impls. Future changes to either type that break compat fail at the wire-compat tests below instead of being caught in a kind-validation cycle.

Wire-compat guard

Four tests in src/handlers/events.rs pin the round-trip semantics:

  • ee4_executor_event_converts_into_event_request — canonical envelope projects into EventRequest with server-only fields taking handler defaults.
  • ee4_event_request_converts_into_executor_event — full-fidelity round-trip back to the envelope.
  • ee4_try_from_event_request_fills_defaults_for_missing_status_and_created_at — when producers omit status / created_at, the conversion applies the same fallbacks the live handler uses (event_status_from_name, Utc::now()).
  • ee4_try_from_event_request_rejects_non_numeric_execution_id — the wire shape is "stringified i64"; anything else surfaces as an error instead of silently dropping the event into the log with execution_id=0.

Conversion impls

impl From<noetl_events::ExecutorEvent> for EventRequest {
    // i64 → String for execution_id / event_id; server-only fields take
    // the same defaults the handler uses when a producer omits them.
}

impl TryFrom<&EventRequest> for noetl_events::ExecutorEvent {
    type Error = anyhow::Error;
    // String → i64 parse can fail (returns 400-equivalent error);
    // missing status / created_at fall back to event_status_from_name /
    // Utc::now() respectively.
}

The conversions are infrastructure — no current call site in the live POST /api/events / POST /api/events/batch handlers consumes them; they're there so follow-up callers can thread the canonical envelope through downstream code without re-deriving the conversion at each site.

Cross-stack envelope state after EE-1 + EE-2 + EE-3 + EE-4 (+ noetl-events crate adoption)

Source event_type field step / node_name field context / payload field worker_id field event_id field
noetl-events::ExecutorEvent (canonical, since 2026-06) event_type step context (alias: payload) optional worker_id optional i64
noetl-executor 0.4.0+ (re-exports from noetl-events) event_type step context (alias: payload) optional worker_id optional i64
noetl-server (Rust) v2.9.0+ (EventRequest + From/TryFrom impls anchored to noetl-events) event_type (alias: name) step payload (alias: context) optional worker_id optional String
Python EventEmitRequest (v3.0.0+) event_type (alias: name) node_name (alias: step) context (alias: payload) optional worker_id optional String
noetl-worker v3.0.0+ event_type step context top-level worker_id None (server gen_snowflake() default fires; app-side generation follow-up)

All four shapes accept the same wire format on input AND emit it on output. The dual-direction alignment lets workers / executors / projectors switch implementations without breaking event ingestion.

Why ExecutorEvent's shape was the target

  • step + status are first-class fields rather than buried in payload — easier for the projector + dashboard queries.
  • created_at is stamped at emit time → avoids server-clock skew when ordering events.
  • execution_id: i64 matches the Postgres bigint column type directly — no String⇄i64 conversion in the ingest path (the wire still uses String to avoid JSON-number precision loss for large snowflakes in browser clients).
  • Documented in executor-crate-architecture as the design target.

Real finding while landing EE-4

Production Python code uses dot-notation event types (step.exit, call.done, command.completed, command.issued, etc.) extensively in noetl/core/dsl/engine/executor/ and noetl/server/api/core/execution.py. But the schema's EventType = Literal["step_completed", "step_started", ...] (underscored, past-tense) only accepted the underscored variants. The strict Literal was either dead code or its validation never actually fired against real worker traffic.

EE-4 loosened EventType to str; semantic validation now lives at the call site (orchestrator + projector) where the real taxonomy is enforced. The dot-notation values are the actual production taxonomy; the Literal was aspirational and never matched reality.

See also

noetl-server

Reference

Design

Operations

Related repos

External

Clone this wiki locally