Umbrella Catalog Model

Umbrella: Catalog model on EHDB

Tracking: noetl/ai-meta#427 · Repo: noetl/catalog · Spec: design/catalog-model.md · Opened: 2026-10-06

A generalized, event-sourced catalog of NoETL's own internal resources — playbooks first — with EHDB as the storage backend.

The one-sentence shape

The catalog is a model and an API, not a datastore: a catalog write becomes an event appended to an EHDB dataset, and every read is a fold over that log. There is no second database, no engine of our own, and no external datastore.

That follows self-sufficiency.md — self-sufficient means NoETL owns its own state; it does not mean no dependencies. ehdb-l0 is a crate that compiles into the binary, which is allowed and preferred. A separate thing to deploy, size, quorum and recover is what is forbidden.

The boundary is load-bearing rather than stylistic: a catalog owning its own storage would be a second source of truth for facts the log already holds, and the two would disagree. That is the failure class representation-drift.md exists to catalogue.

⭐ This finishes a predecessor rather than starting fresh

The single most useful finding of the opening audit, and it changed the project's framing:

component	file	state
CatalogRecord — the event shape	server/src/handlers/catalog_log.rs:89-109	written to its own StoreTier::Catalog, env-gated off (default) | shadow
event types	server/src/handlers/catalog_relation.rs:21-24	catalog.registered / catalog.archived / catalog.restored
CatalogRelation — the fold	server/src/handlers/catalog_relation.rs	pure, IO-free, and nothing calls it on a read path
read-source ladder	server/src/handlers/catalog_read.rs	postgres (default) | verify | tier

catalog_log.rs states the position plainly: "POST /api/catalog/register performs a direct INSERT and there is no emit site anywhere in the catalog service or handler. A relation is by definition a fold of a log, so the catalog relation had nothing to fold." Prod scale: 1,601 records.

So the event vocabulary is inherited, not invented, and the known hazard carries forward: a cached fold is stale by construction for up to the TTL, which is exactly the register-then-run problem. See read-your-writes below.

⚠ The RFCs those files cite — repos/server/docs/rfc/ehdb-catalog-relation.md and …-step3.md — do not exist on disk. Design rationale that was written down has been lost; the spec in noetl/catalog is partly a reconstruction, which is itself a reason it lives there rather than in the server repo.

What the model is

Four entities, and that is the whole model:

ResourceType  ──<  Entity  >──  Attribute
                     │
                     └──<  Relation  >──  Entity

• ResourceType — the type registry (today's noetl.resource). Carries executable, catalogued, an optional attribute_schema, and a supertype.
• Entity — a catalogued resource at one version. Identity is (resource_type, path, version).
• Attribute — typed metadata, the EAV generalization. Text / Integer / Measure / Flag / Timestamp / Ref / Json.
• Relation — a directed edge with a RelationKind and a Provenance.

The generalization, precisely

The reference model (adiona) supplies the entity/attribute/value core and a self-referencing taxonomy, and is deliberately not ported. Its own extensibility stops short: it needs a new table per entity type — item_attributes, trip_attributes, and a trip_category that would need an item_category sibling — so adding a type means DDL.

One polymorphic identity replaces that. Adding a resource type is one appended row: no DDL, no new dataset, no migration, no deploy. AC3 asserts it by checking that the Dataset impl count does not change.

⭐ Demonstrated on a real type, not only asserted. subscription is now supported, and it shares nothing structurally with a playbook:

	playbook	subscription
content	a workflow: tree	a spec:, no workflow: at all
reference location	workflow[].tool[].path	spec.dispatch.playbook (9 of 9 fixtures)
credential dependency	—	spec.auth (6 of 9) → RelationKind::Requires
nesting	arbitrary (iterator, task_sequence)	flat

Supporting it added zero datasets. The playbooks the subscriptions dispatch (sub_ingest_default, handle_webhook) sit in the same fixture directory, so the cross-type graph is closed and checkable rather than asserted.

⭐ spec.auth names a credential, a resource type the catalog deliberately does not hold — the CLI diverts credentials to noetl.credential. The edge is recorded anyway: a dangling dependency is worth knowing about even when the target lives elsewhere.

Also not ported: MySQL triggers, stored procedures, AUTO_INCREMENT, and lang_code localization (NoETL internal resources have no translated names, and an unused column drifts).

Relations — the genuinely new capability

There is no stored relation between catalog entries today. A playbook names a child by path inside a step's tool: block:

tool:
- name: save_profile
  kind: playbook
  path: fixtures/playbooks/playbook_composition/user_profile_scorer

Registration extracts the parent's path, kind, workload:, workflow: and metadata.labels, walks the workflow tree only to validate tool.kind strings, and discards the pairs. So a playbook can be registered referencing a child that does not exist, and nothing says so until the step runs.

The only relation recorded anywhere is execution-level — noetl.execution.parent_execution_id, nullable, no FK. It answers which runs were children of this run, not which playbooks reference this playbook.

Provenance keeps Declared / Extracted / Observed distinct because they are different claims: Extracted says the source says so, Observed says it actually happened. Merging them is the existence-vs-reachability conflation behind a string of false-clean readings in this codebase.

⚠⚠ The trap in the EHDB fold pattern

EHDB has no Projection or Fold trait — 23 public traits workspace-wide, none of them a fold. The pattern is: implement Dataset, then hand-write a …Store whose fold is

self.engine.read_index_after(key, 0)?.into_iter().next_back()

i.e. latest-op-wins per index key. That is correct for the entity log — register → archive → restore is last-write-wins on one path+version — and silently wrong for attributes and relations, where many distinct names and edges coexist under one index key. A naive copy of ProjectionStore keeps only the most recent attribute and returns a plausible-looking row.

AC4 plants two attributes on one entity and asserts both survive. It is written as a positive control precisely because the broken version does not error.

Datasets — a separate namespace

ehdb-l0 has D1–D10, all taken, and ⚠ d7_catalog already exists (CatalogDataset / CatalogOp, keyed by path, mirroring today's flat row).

This repo takes neither D7 nor D11, defining c1..c4 instead:

name	holds	index dim
c1_catalog_entity	register / archive / restore	path
c2_catalog_relation	relation assert / retract	from_path
c3_catalog_attribute	attribute set / unset	path
c4_catalog_type	resource-type declarations	name

Why: Dataset is a public trait and L0Engine<D> is generic over it, so a downstream crate can implement it without touching ehdb-l0; the D-number space belongs to ehdb and taking a number would make this repo's cadence a constraint on theirs; and D7's flat shape is wrong for entity+relation+attribute. Three logs rather than one because the partition key differs — one log keyed by path would make "what references X" a full scan.

Operational requirements EHDB imposes quietly

requirement	why it is not optional
set seal_max_age and drive seal_aged_parts() on a timer	The default None leaves the durability window unbounded in time: a shard that appends a few records and goes quiet never seals, so those records never reach the substrate. ⚠ Setting the field is necessary but not sufficient — the flag is inert on exactly the shard it protects without a timer. The catalog is almost always idle (1,601 records), which makes it the worst case, not a mild one.
FlushPolicy::EveryAppend	CallerDriven is only for a writer that owns its commit points. Catalog writes are rare and matter individually; fsync cost is irrelevant at this volume.
leave manifest_retain at 32	0 means unbounded, which on prod produced 6,770 snapshots / 19.4 GB behind 71.8 MB of data and stopped every append.
pin the same ehdb tag as every process sharing local_root	FORMAT_VERSION is written, matched, or refused. Two versions over one layout is the corruption vector it blocks.
single writer, shard_count = 1	L0Engine takes &mut self and has no built-in mutual exclusion. The only election/fencing code is the shadow-mode, non-authoritative pair in ehdb-reference, which this crate does not depend on.

Read-your-writes

register then immediately execute --path … is the common path — the e2e suite does exactly that in one loop body — so this is the central correctness problem, not an edge case.

EHDB makes it easy and the predecessor's framing slightly pessimistic: L0 reads already merge the active (unsealed) part with local sealed parts and substrate-replica parts, so a read after an append sees the append with no flush. The staleness was a property of the caching layer, not of EHDB.

→ Reads go to read_index_after directly. A cache, if ever added, must be invalidated by op_seq, never by time. AC2 appends then reads with no flush between; that test fails on a TTL cache.

Recent activity

2026-10-06 — reclaim, and a driver that made the rest reachable

catalog#11 — the merge driver was doubling disk. The previous phase proved the part count fell 25 → 4. It did, and bytes rose 42% in the same tick, because the test measured the number the fix was about.

	before	after
live parts	4	4
tick()	merged=3 reclaimed=0	merged=3 reclaimed=48
part files	49 → 56	49 → 8
bytes	94,784 → 189,568 (2.00x)	94,784 → 96,736 (1.02x)

54 part files and substrate objects for 4 live parts. reclaim_orphans is caller-owned and documented as deleting "the superseded source parts a merge leaves behind" — so driving merges without it was a trade, not a fix, reporting success either way. The prod-PVC shape.

⚠ The test's byte assertion was wrong on first draft while the code was right: a merge consolidates parts holding the same records, so bytes are flat by construction. The property that discriminates is that bytes must not grow — 1.02x fixed vs 2.00x broken.

All five caller-owned EHDB lifecycle calls are now accounted for, three driven and two deliberately not:

call	status	why
seal_aged_parts	driven by tick()	seal_max_age is inert without a timer
run_pending_merges	driven by tick()	parts otherwise accumulate monotonically
reclaim_orphans	driven by tick(), after the merges	the manifest swap is what makes sources unreferenced; reclaiming first finds nothing and reports a healthy 0
apply_retention	never	drops whole parts below a floor; the catalog folds latest-op-wins over every op, so a resource whose only Registered op fell below the floor vanishes from latest() while its later attribute ops survive. Correct for an event stream, wrong for a state log
flush_and_wait_uploads	not needed	L0Engine::drop joins the uploader, whose while let Ok(job) = rx.recv() drains the queue before seeing the disconnect (ehdb-l0 engine.rs:654 / :1701)

catalog#12 — nothing could run any of it. Three library crates, zero binaries, and tick() / register_from_source had 0 non-test call sites. 89 tests passed and the catalog had never seen a real document. tick's own doc comment says it must run on a timer; nothing called it once.

The catalog binary (ingest/tick/stats/list/show) reads a git ref, not the working tree — a population sweep on this repo reported adiona yaml: 0 while origin/main carries 53, the checkout being 41 commits behind on a side branch. A bad ref errors rather than returning an empty listing, since an empty listing satisfies "0 skipped" and reads as a clean run.

First real run over the 53:

scanned=53 registered=53 skipped=0 relations=0 attributes=0 kinds={"playbook": 53}

• relations=0 is correct — leaf playbooks calling no child. ⚠ A first regex claimed 53 of 53 carried a child reference; it was matching each document's own metadata.path.
• attributes=0 was a gap: for a playbook find_attributes looked only at metadata.labels, and none of the 53 have labels. All 53 carry a tool kind (53x postgres) and an auth alias (49x adiona_actor, 4x adiona_migrator).

Now uses_tool.<kind> and uses_credential.<alias> → attributes=106, exactly 53 x 2, with the four migrator playbooks cross-checked against ground truth derived independently from git. uses_credential is the one with teeth: rotating adiona_actor means knowing the 49 playbooks that break.

⚠⚠ The alias only, never a value. A scalar auth: is a reference the keychain resolves; a mapping is an inline credential, and copying it would duplicate a secret into a second store. Skipped — and mutation-tested, not trusted: the relaxed check fails the test with the leak in its own output, uses_credential.Mapping {"password": String("hunter2")}. 1 of 5 tests caught it. A templated {{ db_credential }} is skipped too; it names a binding.

99 tests, fmt and clippy -D warnings clean.

date	what	where
2026-10-06	Repo created — public, Apache-2.0, main; protection matches ehdb/server/worker (required check rust, no required review → self-mergeable on green)	noetl/catalog
2026-10-06	CI from commit one, 4 of 4 RED-proven — fmt, clippy -D warnings, tests, source hygiene; each failed on a planted defect, tree restored clean. ⚠ The hygiene guard fired correctly on its first run by refusing to compare against an empty git ls-files	scaffold 089c607
2026-10-06	Design spec merged	catalog#1 → 04a61caa
2026-10-06	Submodule + linkedProjects (also filled the pre-existing repos/ehdb and repos/signal-mesh gaps)	ai-meta#428 → 3d8adf39
2026-10-06	P1 — the four model types, 30 tests, version widened to u32	catalog#2 → e1aae6d8
2026-10-06	⭐ subscription as a real second resource type — 83 tests. The generalization claim stops being structural: AC3 proved it with a synthetic widget, and now proves it with a live type sharing nothing structurally with a playbook (no workflow:, a spec:, its reference at spec.dispatch.playbook in 9 of 9 fixtures, a credential alias at spec.auth). Zero datasets added — RED-proven by planting a c5_mcp_entity. ⭐ RelationKind::Requires finally has a real use	catalog#7 → 5d7f90c2
2026-10-06	P3 — the relation extractor (catalog-extract), 62 tests. Reads both tool: shapes, keeps the target unpinned, provenance Extracted. ⚠⚠ Two testing lessons recorded in code: the negative control needed a second half ("zero references" is also what an unparseable shape returns), and the ORACLE was wrong before the code was — the ground truth counted every document's own kind: Playbook + metadata.path as a self-reference	catalog#5 → 210cf35c
2026-10-06	P2 — the four Dataset impls and their folds, 47 tests. ⚠⚠ The fold was RED-proven three ways: planting the naive last() idiom made the attribute fold return 1 of 4, the relation fold 1 of 3, and the entity fold [1000000] instead of four versions. seal_max_age is set and tick() drives the sealer. AC3 gained a storage half asserting exactly four Dataset impls	catalog#4 → ef1895f1
2026-10-06	Spin-off fix: the kind filter was returning a partial set	server#496 → v3.123.1, #429

Next concrete steps

1. A reverse index for attributes. uses_credential.<alias> is only half its value without "every resource using alias X" — the four datasets index attributes by entity, so that query needs a per-path show today. ⚠ Adding a fifth dataset would violate AC3 (a test pins the Dataset impl count at exactly 4), so the answer is a secondary index key inside c3, not a new dataset. This is a model question, not plumbing.

2. Ingest a second real corpus — the 36 registered muno/* playbooks — to test whether the extractor's relations stays at 0 against documents that do call children. The adiona 53 could not exercise that path at all.

3. Observability. Ticked { sealed, merged, reclaimed } is returned and nothing records it. Per observability.md the three artefacts ship with the change; a catalog running anywhere real needs reclaim to be visible, since its absence is exactly the silent cost #11 was about.

4. ✅ P2 done — AC2, AC4, AC6, AC7 and AC3's storage half discharged. The fold hazard was real and is now guarded: each naive variant was planted and caught.

5. ✅ P3 done — the extractor reads both tool: shapes and emits Extracted relations (AC5). ⚠ Not yet wired to a write path: there is no catalog API to call it from. Emitting on registration arrives with the serving slice.

6. Wire extraction into a write path, so a registration records its edges. Needs the catalog's own API surface, which is the piece neither P1–P3 nor the spec's P4/P5 covers yet.

7. P3 — relation extraction from playbook YAML, Extracted provenance (AC5).

8. P4 — shadow compare against noetl.catalog, reporting coverage and not only divergence. ⚠ A comparator that compares nothing reports perfect agreement; that exact reading closed an issue wrongly before.

9. P5 — serve, once shadow parity holds over a window sized to the period between registration events. ⚠ A clean window shorter than that period is a non-result. Volume is not duration.

Open questions

Flagged in the spec's §13, none blocking:

1. Does noetl.registry merge in? Its own comment records that extending noetl.catalog was considered and rejected, and the rationale is in the RFCs that are missing from disk. The model is a superset — DerivesFrom generalizes its unread lineage column — but the migration is explicitly not performed here.
2. Who owns the DDL during transition? Two byte-identical copies of schema_ddl.sql must be edited together.
3. Is Observed provenance worth a write per run, or should it be sampled?
4. Does the catalog get its own local_root/PVC or share the server's? Shared couples the deploys through FORMAT_VERSION; separate means another volume to size. Leaning shared, since the writer is the server process.

Related

• self-sufficiency.md
• representation-drift.md
• execution-model.md
• Umbrella: EHDB Primary Serve