Skip to content

Claude Handoff EHDB NoETL Integration

Jump to bottom
Kadyapam edited this page Jul 4, 2026 · 1 revision

Claude Handoff: EHDB Development And NoETL Integration

Status date: 2026-07-04 UTC

This page is the handoff brief for continuing EHDB as the NoETL-specialized Event Horizon Database and integrating it into the NoETL ecosystem. It is written for Claude or another coding agent that does not have the full chat history.

Mission

EHDB is not a generic database first. It is the NoETL-domain storage substrate intended to collapse the platform roles currently served by PostgreSQL, NATS JetStream, external object stores, Qdrant, and ClickHouse into one NoETL-centric durable fabric.

EHDB must stay aligned with the NoETL execution model:

gateway = gatekeeper
worker = atomic compute
playbook = ephemeral blueprint
shared cache = state vehicle
event log = source of truth

Gateway/API/server roles may embed EHDB for control-plane planning and validation. They must not execute local-reference data-plane helpers or directly touch EHDB storage. Worker, playbook, and system roles may receive explicit data-plane capabilities for bounded work.

Repositories And Current Pointers

Current merged code anchors:

  • noetl/ehdb: 0dc2016f4b692d3d868ccbc3918900962a880ca1 (ehdb-local-reference summary --log <path>)
  • noetl/noetl: f82ccb7f8fb2c2b8ccbc9881ca230d0ca60a8edd (typed EHDB local-reference readiness summary)
  • ai-meta: should track the merged submodule SHAs and memory notes after every upstream merge.

Implemented EHDB Foundation

The local reference implementation already includes:

  • Rust workspace crates: ehdb-core, ehdb-catalog, ehdb-storage, ehdb-stream, ehdb-retrieval, ehdb-system, ehdb-transaction, ehdb-reference, and ehdb-service.
  • Catalog-as-database metadata with table schemas, immutable snapshots, scan grants, strict JSON decode, identifier validation, and duplicate column rejection.
  • Content-checked immutable object references with byte length, SHA-256 digest, geo placement, and data-gravity shard pointers.
  • Placement policy, replication planning, durable replica inventory, and bounded local replication executor metadata.
  • Local Arrow IPC table write/read fixtures, latest snapshot scanner, projection, equality filters, and Arrow Flight ticket/result/info codecs.
  • Local Arrow Flight service/server trait adapter, loopback listener, auth header policy, tenant/namespace scope guard, scan grant enforcement, bounded request concurrency, and redacted access-log policy.
  • Local JSONL transaction log, stream journal, and system-library journal with replay validation.
  • Stream model with subjects, subject filters, durable consumers, retention, ack cursors, and strict replay decode.
  • RAG/retrieval model with documents, chunks, embeddings, exact vector search, text search, hybrid search, bounded retrieval-context assembly, versioned payload codecs, receipt payloads/events, and caller-controlled receipt stream helpers.
  • System WASM library manifests and environment/channel bindings for hot-replaceable system playbook functionality.
  • Embedded NoETL role/capability policy that keeps gateway/API roles control-plane-only while allowing bounded worker/playbook/system data-plane capabilities.
  • ehdb-local-reference summary --log <path>, a bounded diagnostic helper that replays a local JSONL transaction log and emits deterministic counts across transaction, catalog, stream, retrieval, system-library, and storage domains.

Implemented NoETL Integration

NoETL currently has the side-effect-free integration envelope:

  • noetl.core.ehdb_contract: feature-gated EHDB env contract, disabled by default.
  • noetl.core.ehdb_control_plane: gateway/API/server control-plane-only descriptor.
  • noetl.core.ehdb_adapter: worker/playbook local-reference adapter, helper discovery, bounded subprocess execution, and typed LocalReferenceEhdbSummary.
  • noetl.core.ehdb_surface: selects control-plane or local-reference surface from env without opening logs or touching storage.
  • scripts/smoke_ehdb_local_reference_summary.py: end-to-end smoke against ehdb-local-reference summary.
  • NoETL dev and pip Dockerfiles build ehdb-local-reference from pinned EHDB ref 0dc2016f4b692d3d868ccbc3918900962a880ca1, copy only the binary into the final runtime image, and fail the build if ehdb-local-reference --help cannot run.

Important NoETL PRs:

Tracking Issues And Phase Links

Core open umbrellas:

Recent closed implementation anchors:

Use the Roadmap page for older phase details: https://github.com/noetl/ehdb/wiki/Roadmap

Next Development Phases

Phase A: Ops And Runtime Enablement

Goal: make the packaged helper available through NoETL deploy/runtime configuration without changing the default disabled behavior.

Implementation notes:

  • Work primarily in noetl/ops and, if needed, noetl/noetl.
  • Add disabled-by-default Helm/local deploy values for EHDB.
  • Render role-specific env:
    • server/gateway/API: NOETL_EHDB_ENABLED=true, NOETL_EHDB_MODE=control_plane, NOETL_EHDB_CLIENT_ROLE=server|api|gateway, NOETL_EHDB_CAPABILITIES=control_plane
    • worker/playbook/system: NOETL_EHDB_ENABLED=true, NOETL_EHDB_MODE=local_reference, NOETL_EHDB_CLIENT_ROLE=worker|playbook|system, NOETL_EHDB_LOCAL_REFERENCE_LOG=/opt/noetl/data/ehdb/local-reference.jsonl
  • Keep the feature off by default. Enabling should be explicit in a values overlay or playbook input.
  • Validate local kind before any GKE rollout.

Tests:

  • helm template checks that disabled defaults render no EHDB env.
  • helm template checks that enabled worker env includes local reference settings and server env remains control-plane-only.
  • Kind Job or pod smoke runs python scripts/smoke_ehdb_local_reference_summary.py inside the NoETL image with imagePullPolicy: Never.

Phase B: Worker/Playbook Readiness Hook

Goal: expose a bounded worker/playbook preflight that confirms EHDB local-reference readiness without gateway data access.

Implementation notes:

  • Reuse read_ehdb_local_reference_summary_from_env.
  • Prefer a worker/playbook-local command or smoke step, not a server endpoint.
  • If a server/admin surface is needed later, it must be control-plane-only and must not execute the helper.

Tests:

  • Disabled env returns no summary and no side effects.
  • Worker/playbook env executes helper and validates typed summary.
  • Gateway/API/server local-reference env is rejected.
  • Missing helper produces a clear validation error.

Phase C: EHDB Event Stream Integration Path

Goal: define the migration path from NATS JetStream-backed NoETL command and event streams toward EHDB-native streams.

Implementation notes:

  • Start with local reference adapters and bridge tests.
  • Preserve NATS as a migration adapter, not the target product boundary.
  • Do not put stream append/consume logic in gateway.
  • Keep durable consumers and ack-after-materialize semantics explicit.

Tests:

  • Publish/replay/ack/consumer cursor restart tests in EHDB.
  • NoETL worker/playbook smoke showing an EHDB stream fixture can represent command/event semantics.
  • Duplicate, malformed subject, wildcard, retention, and replay validation.

Phase D: EHDB System WASM Store Integration

Goal: move NoETL system playbook library resolution toward EHDB-owned system library manifests and environment/channel bindings.

Implementation notes:

  • Keep WASM execution in worker/system-pool roles.
  • EHDB stores manifests, object refs, digests, host capability requirements, and active channel bindings.
  • Support hot replacement by rebinding channel/environment to a new digest/revision.

Tests:

  • Publish/bind/resolve across tenant, namespace, environment, channel, and logical path.
  • Rebinding preserves older immutable manifests.
  • Worker/system role can resolve and execute a fixture module reference.
  • Gateway/API cannot execute or data-touch the system library store.

Phase E: RAG Retrieval Integration

Goal: replace external vector-store dependency paths with EHDB-native RAG document/chunk/embedding metadata and bounded retrieval context assembly.

Implementation notes:

  • Keep retrieval execution in worker/playbook roles.
  • Use EHDB retrieval payload codecs and receipt/event shapes.
  • Avoid raw embeddings or prompt content in logs/receipts.

Tests:

  • Register document/chunk/embedding metadata.
  • Vector, text, and hybrid search fixtures.
  • Bounded context assembly with byte/text limits.
  • Receipt payload and stream event validation.
  • Scope guard rejects cross-tenant/namespace requests.

Phase F: Distributed Durability And Replication

Goal: move beyond local JSONL references toward replicated metadata and distributed storage placement.

Implementation notes:

  • Keep consensus behind transaction-log traits.
  • Use existing geo placement and data-gravity shard metadata for planning.
  • Replication execution should be bounded worker/playbook/system work, not gateway work.

Tests:

  • Consensus/log trait contract tests.
  • Replay and crash/restart compatibility.
  • Placement policy and replica inventory tests.
  • Copy-needed plan execution with digest/length verification.

Phase G: Analytical Query Path

Goal: evolve the Arrow scan/Flight fixtures toward a NoETL analytical read path.

Implementation notes:

  • Keep current Arrow Flight loopback and service trait contracts.
  • Add SQL/planner/predicate-pushdown only after metadata and scan contracts are stable.
  • Gateway remains an admission/control-plane surface, not a direct reader.

Tests:

  • Arrow schema, FlightInfo, Ticket, and FlightData validation.
  • Projection/predicate validation and decoded row checks.
  • Auth header, tenant/namespace scope, scan grant, access-log, and concurrency tests.

Architectural Decisions To Preserve

  • EHDB is NoETL-specialized, not generic database-first.
  • The catalog lives inside EHDB.
  • Transaction log/event log is the source of truth; caches are derived.
  • Gateway/API/server roles are control-plane-only for EHDB.
  • Data touch belongs in bounded worker/playbook/system steps.
  • No persistent per-tenant AI-agent/MCP-server processes are required.
  • System WASM libraries are hot-replaceable by digest/revision binding, not crate version churn.
  • Geo placement and data-gravity shard pointers are storage metadata for future routing, replication, compaction locality, and read scheduling.
  • Local JSONL logs and loopback services are reference fixtures, not production distributed durability.
  • Container/image changes must be validated in local kind before GKE.

Validation Commands

EHDB code:

cargo fmt --all --check
cargo test --workspace
cargo clippy --workspace --all-targets -- -D warnings
cargo bench --workspace --no-run
cargo test -p ehdb-stream -p ehdb-retrieval -p ehdb-reference

NoETL EHDB integration:

.venv/bin/python -m pytest \
  tests/core/test_ehdb_contract.py \
  tests/core/test_ehdb_adapter.py \
  tests/core/test_ehdb_control_plane.py \
  tests/core/test_ehdb_surface.py \
  tests/scripts/test_smoke_ehdb_local_reference_summary.py

.venv/bin/python -m pytest \
  tests/core/test_ehdb_contract.py \
  tests/core/test_ehdb_adapter.py \
  tests/core/test_ehdb_control_plane.py \
  tests/core/test_ehdb_surface.py \
  tests/core/test_runtime_topology.py \
  tests/core/runtime/test_pool_routing.py \
  tests/scripts/test_smoke_ehdb_local_reference_summary.py

.venv/bin/python -m compileall -q \
  noetl/core/ehdb_contract.py \
  noetl/core/ehdb_adapter.py \
  noetl/core/ehdb_control_plane.py \
  noetl/core/ehdb_surface.py \
  scripts/smoke_ehdb_local_reference_summary.py

.venv/bin/python scripts/smoke_ehdb_local_reference_summary.py \
  --helper-bin ../ehdb/target/release/ehdb-local-reference \
  --log /tmp/noetl-ehdb-smoke.jsonl

NoETL image/kind gate:

env -u XDG_DATA_HOME podman build --platform linux/arm64 \
  -t local/noetl:ehdb-helper-image-test \
  -f docker/noetl/dev/Dockerfile .

env -u XDG_DATA_HOME podman run --rm \
  local/noetl:ehdb-helper-image-test \
  python scripts/smoke_ehdb_local_reference_summary.py \
  --log /tmp/noetl-ehdb-image-smoke.jsonl

For kind, load the exact image into kind-noetl and run a one-off Job with imagePullPolicy: Never before any GKE deployment.

Claude Kickoff Prompt

Use this prompt in Claude chat:

You are continuing EHDB development for the NoETL ecosystem.

Read first:
- https://github.com/noetl/ehdb/wiki/Claude-Handoff-EHDB-NoETL-Integration
- https://github.com/noetl/ehdb/wiki/Architecture
- https://github.com/noetl/ehdb/wiki/Roadmap
- https://github.com/noetl/ehdb/issues/1
- https://github.com/noetl/ehdb/issues/2
- https://github.com/noetl/ehdb/issues/3
- https://github.com/noetl/ehdb/issues/4
- https://github.com/noetl/ehdb/issues/5
- https://github.com/noetl/ehdb/issues/6

Working model:
- ai-meta is the orchestration repo; product code belongs in submodules.
- EHDB is NoETL-specialized storage, not a generic database first.
- Preserve the NoETL execution model:
  gateway = gatekeeper, worker = atomic compute, playbook = ephemeral blueprint,
  shared cache = state vehicle, event log = source of truth.
- Gateway/API/server EHDB embedding is control-plane-only. Do not add direct
  gateway/API/server data-plane storage access.
- Worker/playbook/system roles may use bounded EHDB data-plane capabilities.
- No persistent per-tenant agent/MCP service should hold state between requests.
- Validate container/image changes in local kind before GKE.

Start with Phase A from the handoff page: ops/runtime enablement.
Create or reuse a GitHub issue, branch from the appropriate repo main,
add disabled-by-default Helm/local deploy EHDB configuration, render
role-specific env safely, test disabled defaults and enabled worker
local-reference mode, run the NoETL EHDB smoke in a local kind Job, then
open and merge a PR. After merging, update ai-meta memory and submodule
pointers.

Report back with:
- issue/PR/wiki links,
- exact commits,
- validation commands and results,
- any architectural decision that needs to be added to the wiki.

Handoff Checklist For Next Agent

  1. Sync submodules from ai-meta.
  2. Confirm repos/ehdb, repos/noetl, and repos/ehdb-wiki are clean.
  3. Pick one phase from this page, preferably Phase A.
  4. Open or update a GitHub issue before coding.
  5. Update the EHDB wiki with any design change in the same slice.
  6. Implement in the product submodule, not in ai-meta.
  7. Run focused tests, then nearby/regression tests.
  8. For image/deploy changes, validate in local kind-noetl using Podman, not Docker/Colima fallback.
  9. Merge upstream PR.
  10. Update ai-meta submodule pointer and memory.

EHDB

Programs

Kind validation

Query interface

Runbooks

Design notes

GitHub

Clone this wiki locally