EHDB Query Interface

The EHDB Data Query Interface is the read-only query surface over NoETL platform data — executions, events, and derived state that NoETL owns. It has two halves:

a read-only HTTP API on the noetl server under /api/ehdb/...;
a noetl ehdb query ... CLI console that is a thin client of that API.

Tracked by noetl/ai-meta#178; first slice landed via noetl/server#277 + noetl/cli#61. It is the read counterpart to the write/serve path built across Phases 6–10 (see the Roadmap).

Live in the kind stack (2026-07-06)

The projection/read-model surface is now running live end-to-end in the local kind-noetl stack (podman provider, LOCAL only — not GKE, not prod). The full path is wired: a real drive lands events in noetl.event, the worker's EHDB event-log shadow (worker v5.67.0, NOETL_EHDB_EVENTLOG=shadow on both data-plane pools) mirrors those events into the reference tier, and the noetl server v3.53.0 (carrying /api/ehdb/*) serves them back through noetl ehdb query.

Proof — the mirrored drive execution 332760742153424896 (automation/pft_sql_probe_v2, COMPLETED, 13 events) surfaced through every endpoint:

GET /api/ehdb → 200 (manifest, read_only:true, control_plane:true).
GET /api/ehdb/executions → 200, list includes 332760742153424896.
GET /api/ehdb/executions/332760742153424896 → 200, derived state (status=COMPLETED, current_node=playbook, event_count=13).
GET /api/ehdb/executions/332760742153424896/events → 200, read-model rows (projected columns only).
GET /api/ehdb/events → 200 (global scan).
GET /api/ehdb/tiers/{eventlog,kv} → 501 with the routing contract + control_plane_guard — the raw-tier seam is documented, not yet wired.

The CLI (noetl ehdb query executions | execution <id> | events <id>, table + --json, built from cli v4.12.0) returns the same real rows against the in-cluster server. Deploy shape stays control-plane: the server carries no NOETL_EHDB_* env and opens no data-plane tier storage — it serves read-models directly from noetl.event and routes raw tiers to the worker. Reversible: roll the server image back to the prior localhost/noetl-server:auth-sync-167 to undo.

Scope + guarantees

Read-only. No endpoint or subcommand mutates. There are no write / delete / update routes in this surface.
Platform-only. Queries NoETL's own operational data (executions, events, state) — the noetl.* read-model. It never reaches business data, which lives in external stores accessed through playbook connectors.
Secret-free by construction. Every response carries only projected read-model columns — ids, status, node names, counts, timestamps, paths. The result / error / context / workload payload bodies (which can carry credential material) are never selected or surfaced, mirroring the payload-free ExecutionStateView / EventReadModelView discipline. Secret-free is structural, not a post-hoc scrub.
Bounded. Every list/scan is capped server-side (limit clamped to ≤ 1000) with a forward after cursor for pagination.

Control-plane routing design (load-bearing)

The server stays control-plane. It splits query handling in two:

Query class	How it is served
Projection / read-model (executions list, execution state, event read-model)	Served directly from the read-model store the server already reads (Postgres `noetl.event`), via the existing `ExecutionService`.
Raw data-plane tier (raw event-log scan, KV, object, vector)	Relayed to the worker data-plane via a direct synchronous HTTP read to the worker query port (`worker-service:9090`). The server does not open tier storage itself, and the read does not ride the NATS drive/command bus.

The load-bearing rule: the server never embeds the EHDB data-plane engine. No EHDB tier storage is opened in the server process — the same guard the worker enforces (worker/src/ehdb/guard.rs, roles server / api / gateway refused data-plane access) holds on the server side by construction, because the engine simply isn't linked in. This keeps the loose-coupling boundary from the RFC intact: the control plane gatekeeps and read-serves the read-model; the data plane owns tier storage.

Why a direct data-plane call, not the drive. A query is a read, not a unit of playbook work. Enqueuing it on the NATS drive would make a read wait behind the command queue and hold a worker claim slot for a filesystem scan. Instead the server makes a synchronous HTTP GET straight to the worker's existing metrics/query port (:9090) — the worker resolves the request under its data-plane guard, opens the same tier driver the mirror wrote through, reads, and returns the tier *Outcome. The drive stays for work; reads take the data-plane read path.

noetl ehdb query ...            (CLI, thin read-only client)
        │  GET /api/ehdb/...
        ▼
   noetl server (control plane)
        ├── projection/read-model ──► Postgres read-model     (served DIRECT)
        └── raw data-plane tier ─────► HTTP GET worker-service:9090  (DIRECT, not the drive)
                                          │  /ehdb/tiers/{tier}
                                          ▼
                                       noetl worker (data plane)
                                          └── ehdb_reference tier drivers

Server API reference

Base namespace: /api/ehdb. All routes are GET. Auth follows the same posture as the sibling /api/executions routes — the gateway is the auth-enforcement point in front of the server; this surface exposes the read-model read scope only.

`GET /api/ehdb`

Interface manifest — advertises the endpoints and the per-tier routing status (served-direct vs routed). Useful for discovery.

`GET /api/ehdb/executions`

List executions from the projection read-model.

Query params: status, path, catalog_id, limit (default 50, max 1000), offset.

{
  "action": "ehdb.executions.list",
  "tier": "projection",
  "limit": 50, "offset": 0, "returned": 2,
  "executions": [
    {
      "execution_id": "1234567890", "catalog_id": "42",
      "path": "weather/forecast", "status": "COMPLETED",
      "terminal": true, "current_node": null,
      "parent_execution_id": null, "event_count": 7,
      "started_at": "2026-07-06T18:00:00Z",
      "completed_at": "2026-07-06T18:00:12Z"
    }
  ]
}

current_node / parent_execution_id are null in the list projection (not loaded by the bounded list aggregation) — fetch the single execution for those.

`GET /api/ehdb/executions/{execution_id}`

One execution's derived-state read-model. status reuses the server's existing terminal-priority derivation. Absent executions return exists: false (read-model semantics), not a 404.

{
  "action": "ehdb.execution.state",
  "tier": "projection",
  "execution_id": "1234567890", "exists": true,
  "state": {
    "execution_id": "1234567890", "catalog_id": "42",
    "path": "weather/forecast", "status": "COMPLETED",
    "terminal": true, "current_node": "end",
    "parent_execution_id": null, "event_count": 7,
    "started_at": "...", "completed_at": "..."
  }
}

`GET /api/ehdb/executions/{execution_id}/events`

Event read-model scoped to one execution. Ordered ASC by event_id (the application-side snowflake, the monotonic global-ordering key).

Query params: limit (default 100, max 1000), after (forward cursor: event_id > after).

{
  "action": "ehdb.execution.events",
  "tier": "projection",
  "execution_id": "1234567890", "exists": true,
  "limit": 100, "returned": 3, "next_cursor": null,
  "events": [
    {
      "event_id": "1234567891", "execution_id": "1234567890",
      "event_type": "playbook.started", "node_name": "start",
      "status": "RUNNING", "created_at": "..."
    }
  ]
}

`GET /api/ehdb/events`

Event read-model scan by global sequence (across the log). Same params + shape as the per-execution events endpoint, plus an execution_id on each row. Served from the cluster-master pool; in single-pool (kind / unsharded) deployments this is the whole log and globally ordered. A globally-ordered scan under multi-shard prod needs a per-shard fan-out + k-way merge (see Remaining slices).

`GET /api/ehdb/tiers/{tier}` — raw data-plane tier query (LIVE)

tier ∈ eventlog | kv | object | vector. The server relays the read to the worker data-plane and returns the worker's tier *Outcome verbatim. The flow:

Server validates tier and forwards every query-string param to the worker unchanged.
Server makes a synchronous HTTP GET to ${NOETL_EHDB_WORKER_QUERY_URL}/ehdb/tiers/{tier}?<params> (the worker metrics/query Service on :9090). No drive enqueue.
Worker guards the request (assert_data_plane_access_allowed — refuses server/api/gateway roles), opens the same tier driver the mirror writes through, and invokes its read method: EventLogDriver::scan_global / read_execution, KvStateDriver::get / scan, ObjectBlobDriver::get / list / locate, VectorDriver::query.
Worker returns the tier *Outcome (already Serialize + secret-free) wrapped in {action, tier, op, outcome, result}; the server relays that body + HTTP status.

Disabled by default: the handler is a strict no-op (outcome=disabled, 200) unless NOETL_EHDB_ENABLED is truthy on the worker. Bounded: every list/scan clamps limit (and vector top_k) to ≤ 1000. Outcome → status map: served/absent/disabled → 200, rejected/invalid → 400, guard_refused → 403, unavailable → 503; a relay-not-configured server (no NOETL_EHDB_WORKER_QUERY_URL) → 501.

Query params by tier

Tier	Params
`eventlog`	`execution` (per-execution ordered read; omit for a global scan), `after`, `limit`
`kv`	`bucket` (required), `key` (single-key `get`; omit for a bucket `scan`), `prefix`, `limit`
`object`	`op` (`get`\|`list`\|`locate`; default `get` with `key`, else `list`), `key`, `prefix`, `limit`
`vector`	`collection` + `model_id` + `vector` (comma-separated floats) — all required, `top_k`

Common to all: tenant, namespace (default the reference tenant/namespace), execution_id (correlation only — echoed into the worker's ehdb.query span, never a metric label).

Sharded event-log scan. When the worker runs the durable event-log backend (NOETL_EHDB_EVENTLOG_BACKEND=durable_segment), an eventlog global scan fans out a k-way merge across every per-shard segment store this replica owns, ordered by global_sequence (ties broken by execution_id). Unsharded (kind default), it degrades to the single log. The merge hot path (worker/src/ehdb/query.rs::merge_shard_records) is pure + benched (~4.2 M records/s merging 8×4096).

Observability. Each query records noetl_worker_ehdb_query_ops_total{tier,operation,outcome} + noetl_worker_ehdb_query_duration_seconds on the worker /metrics surface (disabled → no line, preserving the byte-identical-when-off invariant), and rides an ehdb.query tracing span carrying execution_id.

CLI usage

noetl ehdb query ... mirrors the endpoints. Table output by default; --json for the raw response. The CLI resolves the server URL, gateway-proxy mode, and session token exactly like the other noetl subcommands.

# List executions (optionally filtered / paginated)
noetl ehdb query executions
noetl ehdb query executions --status RUNNING --limit 20 --offset 0
noetl ehdb query executions --json

# One execution's derived state
noetl ehdb query execution 1234567890
noetl ehdb query execution 1234567890 --json

# The event read-model for an execution
noetl ehdb query events 1234567890
noetl ehdb query events 1234567890 --limit 200 --after 1234567900 --json

# Raw data-plane tiers (relayed to the worker; secret-free)
noetl ehdb query tier eventlog --limit 20
noetl ehdb query tier eventlog --execution 1234567890
noetl ehdb query tier kv --bucket noetl-state --prefix exec/
noetl ehdb query tier kv --bucket noetl-state --key exec/123/status
noetl ehdb query tier object --prefix noetl/ --limit 50
noetl ehdb query tier object --key noetl/123/state --op locate
noetl ehdb query tier vector --collection c --model-id m --vector 0.1,0.2,0.3 --top-k 5

Example table:

┌────────────┬───────────┬──────────────────┬────────┬──────────────────────┐
│ execution_id │ status    │ path             │ events │ started_at           │
├────────────┼───────────┼──────────────────┼────────┼──────────────────────┤
│ 1234567890 │ COMPLETED │ weather/forecast │ 7      │ 2026-07-06T18:00:00Z │
└────────────┴───────────┴──────────────────┴────────┴──────────────────────┘
(1 rows)

Raw data-plane tiers live (2026-07-10)

The worker-side query handler landed (worker/src/ehdb/query.rs), turning the /api/ehdb/tiers/{tier} seam from a 501 stub into a live relay. The server's raw_tier_query now makes the direct HTTP read to the worker query port (NOETL_EHDB_WORKER_QUERY_URL, defaulted in ops to http://noetl-worker-rust-metrics:9090); the CLI gained noetl ehdb query tier <eventlog|kv|object|vector>. All four tiers serve their ehdb_reference driver reads under the data-plane guard, disabled-by-default, bounded, secret-free. The durable event-log backend adds the sharded k-way merge for the raw eventlog scan.

Pod-local caveat (kind/shadow). The mirror writes the local_reference log (and durable segments) to pod-local storage. The worker query Service is headless, so for coherent reads point NOETL_EHDB_WORKER_QUERY_URL at a single-replica pool (or one whose shards are hydrated). This matches the shadow / kind validation posture; multi-replica coherent reads are a prod concern tracked with the durable shared-tier + affinity work.

Remaining slices

Done: the projection/read-model surface (first slice) and the four raw data-plane tiers + the sharded event-log merge (this slice). Still open:

Sharded global scan for the read-model GET /api/ehdb/events — the server's Postgres-backed global scan is still single-pool; the raw tier=eventlog scan already does the durable per-shard fan-out + k-way merge, but the projection read-model scan does not.
Multi-replica coherent reads — the query targets one pod's local mirror (see the pod-local caveat above); a coherent read across a multi-replica pool needs the durable shared-tier hydrate path, not just a Service round-robin.

Projection / Read-Model Engine — the read-model views this API mirrors.
Event-Log Core Engine — the raw event-log tier behind the routing seam.
Backend Configuration — per-tier backend resolution the worker uses when serving routed queries.
RFC: Completion Program — the loose-coupling boundary this surface honours.
Roadmap — phased plan.

EHDB Query Interface

EHDB Query Interface

Live in the kind stack (2026-07-06)

Scope + guarantees

Control-plane routing design (load-bearing)

Server API reference

GET /api/ehdb

GET /api/ehdb/executions

GET /api/ehdb/executions/{execution_id}

GET /api/ehdb/executions/{execution_id}/events

GET /api/ehdb/events

GET /api/ehdb/tiers/{tier} — raw data-plane tier query (LIVE)

CLI usage

Raw data-plane tiers live (2026-07-10)

Remaining slices

Related

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

EHDB

Programs

Kind validation

Query interface

Runbooks

Design notes

GitHub

Clone this wiki locally

`GET /api/ehdb`

`GET /api/ehdb/executions`

`GET /api/ehdb/executions/{execution_id}`

`GET /api/ehdb/executions/{execution_id}/events`

`GET /api/ehdb/events`

`GET /api/ehdb/tiers/{tier}` — raw data-plane tier query (LIVE)