Umbrella System Pool Design

Umbrella — System Worker Pool + Compilable Playbook Plug-ins (PRIMARY)

ai-task: noetl/ai-meta#46 · Opened: 2026-06-02 · Promoted to primary migration umbrella: 2026-06-02 (afternoon — same day) · Last update: 2026-06-02 · Status: Design done (ADR v2 landed); Phase 1 + the first system playbook (system/outbox_publisher) ready to pick up · ADR (v2): System Worker Pool and WASM Plug-in Surface · Closed predecessors: #30 Appendix H worker migration, #45 compiled Python rewrite

Primary migration umbrella as of 2026-06-02 afternoon. Rest of the Python→Rust migration goes through this umbrella, not via more compiled binaries. The platform extends itself via its own primitives: compilable + pluggable system playbooks on a privileged worker pool.

Goal

Build the system worker pool — a privileged Rust worker pool that runs platform-internal logic (auth, RBAC, scheduled cleanups, AND publisher + projector) as NoETL playbooks under a system/ namespace. WASM is the compilation target so playbooks are hot-reloadable AND tenant-overridable.

Model analogy: Oracle's SYS schema (privileged namespace, platform extends itself with its own primitives) plus PostgreSQL extensions (CREATE EXTENSION loads compiled code at runtime via dlopen).

The big idea

Instead of writing more compiled Rust services, encode services as playbooks. Three properties land for free:

1. Audit + replay — every system action emits events the same way user actions do; the event log covers system actions too.
2. Versioning — catalog entries are versioned; system playbook changes are tracked in git AND in the catalog.
3. Tenant override — tenants can supply <tenant>/system/<name> to customise platform behavior for their tenant without forking the platform.

WASM compilation makes the perf argument moot — a YAML playbook compiled to WASM runs at near-native speed inside the worker's wasmtime host.

Phases

Phase	What	Status
1	System worker pool runtime	Not started; ready to pick up
2	Migration system playbooks (publisher + projector)	Not started; depends on Phase 1
3	Additional system playbooks (auth, RBAC, cleanup)	Not started; depends on Phase 1
4	YAML → WASM compilation pipeline + hot reload	Not started; perf optimisation

Phase 1 — System worker pool runtime

The system worker pool is NOT a new compiled binary. It reuses the existing noetl/worker Rust binary with distinct configuration. Sub-tasks:

• Helm Deployment noetl-worker-system-pool — image ghcr.io/noetl/worker:<v>, distinct ServiceAccount, env NATS_CONSUMER=noetl_worker_pool_system + NATS_FILTER_SUBJECT=noetl.commands.system.>.
• KEDA ScaledObject — smaller cap (5) and tighter lag threshold (5) than user pools. Pre-sketched at noetl-ops wiki — System worker pool.
• POOL_FILTER_MAP extension in repos/noetl/noetl/core/runtime/pool_routing.py — system_* tool kinds route to the system pool segment.
• Server-side validation — only catalog entries under system/ path may declare system_* tool kinds.
• RBAC for the system pool's ServiceAccount — read keychain, write noetl.event, mutate catalog.
• Helm workerPools.system values section (opt-in, default disabled).
• Kind validation rig — repos/ops/automation/development/system-pool-validation.yaml + validate-system-pool.sh.

Acceptance: noetl-worker-system-pool deployment scales 1→N on noetl_worker_pool_system consumer lag; commands published to noetl.commands.system.<eid> get claimed by the system pool and not by user pools.

Phase 2 — Migration system playbooks (retire Python pods)

These playbooks REPLACE today's Python pods. Each one is pure YAML using existing tool kinds.

• system/outbox_publisher — replaces python -m noetl.outbox_publisher. Steps:

  1. tool: postgres — claim outbox batch (SELECT outbox_id FROM noetl.outbox WHERE status IN ('PENDING', 'FAILED') ... FOR UPDATE SKIP LOCKED, mark IN_FLIGHT, attempts+1).
  2. tool: nats (iterator over batch rows) — publish JSON payload to noetl.events.<tenant>.<org>.<eid>.<shard>.
  3. tool: postgres — mark rows PUBLISHED on success, FAILED with exponential backoff on error.

  Triggered by KEDA on outbox depth OR by NATS system/outbox.tick subject from a server-side LISTEN/NOTIFY bridge (TBD).

• system/projector — replaces python -m noetl.projector. Steps:

  1. tool: nats — subscribe to noetl.events.> with consumer name = system-projector-<shard_id> (derived from worker pod name).
  2. tool: postgres — batch INSERT into noetl.event with ON CONFLICT (event_id) DO NOTHING for idempotency.
  3. Ack the NATS batch.

  Sharded — one playbook execution per shard, shard derived from worker_id.

Acceptance: the Python noetl-outbox-publisher Deployment and noetl-projector StatefulSet retire from the kind cluster + GKE. Outbox throughput + projection lag match or exceed today's Python baseline.

Phase 3 — Additional system playbooks (pluggable surface)

The interesting part — once the runtime + first two playbooks land, the rest of the platform's internal logic moves to playbooks too:

• system/auth — session validation, token lookup, IdP integration. Tenant override: <tenant>/system/auth_with_saml, <tenant>/system/auth_with_oidc, etc.
• system/rbac — per-action authorisation. Tenant override supported.
• system/scheduled_cleanup — TTL enforcement, stale-row reaping. Scheduled via cron-style tool kind.
• system/credential_rotate — refresh long-lived tokens before expiry.
• Custom dispatcher rules — tenant-supplied logic for routing decisions, e.g. "route all mcp calls for tenant X to pool Y".

Phase 4 — Compilation (perf + plug-in surface)

The WASM compilation pipeline that closes the perf gap for hot-loop system playbooks (projector, publisher):

• YAML → WASM compiler — server-side at catalog register time (or first execute). Output cached by (path, version, digest).
• wasmtime host inside noetl/worker — capability-based imports per the ADR. Different capability sets for system vs. tenant-supplied modules.
• Hot reload — catalog version bump invalidates cache; next claim recompiles + loads. No process restart.
• WasmPlaybook catalog kind — first-class catalog entry type (per the ADR's Option 1; Option 2 is the future "YAML stays source; WASM is internal" optimisation).

Sequencing — Step 1 cut

Step 1 = Phase 1 + system/outbox_publisher playbook (Phase 2 first half). Minimum cut that retires the Python outbox publisher pod.

Why this cut: validates the full pipe end-to-end (NATS routing → system pool worker → playbook execution → tool dispatch → Postgres + NATS effects) with the smallest scope. Once it works, adding more system playbooks is incremental.

After Step 1 lands:

• Step 2: system/projector (retires the second Python pod).
• Step 3: system/auth (the first non-migration system playbook).
• Step 4: WASM compilation pipeline.

What's NOT in this umbrella

• HTTP server FastAPI parity in Rust — separate concern, separate future umbrella if needed.
• Rust worker tool-kind parity gaps (#47, #48) — those stay in Umbrella: Rust Worker Parity Gaps.
• Container tool kind callback design (#43) — separate concern.

Recent activity

Date	Event
2026-06-02 (morning)	Issue filed during the placement discussion under #45
2026-06-02 (morning)	Hot-reload trade-off matrix (libloading / WASM / sub-process / closure JIT) captured as comment on #46
2026-06-02 (morning)	ADR v1 merged via noetl/docs#176 — publisher + projector classified as compiled core
2026-06-02 (morning)	Cross-linked wiki pages live: noetl-server runtime-shape (capability surface) and noetl-ops system-worker-pool
2026-06-02 (afternoon)	noetl/server#10 opened as #45 step 1 (Rust publisher binary)
2026-06-02 (afternoon)	Architecture pivot: user decision that publisher + projector belong as system playbooks, not compiled binaries. #30 closed, #45 closed, noetl/server#10 closed, this umbrella promoted to primary. ADR v2 in flight to reflect corrected classification.
2026-06-02 (afternoon)	No implementation started. Phase 1 + system/outbox_publisher ready to pick up.

Next concrete steps

1. Land ADR v2 — corrects the classification (publisher + projector → plug-in ring). PR open on noetl/docs.
2. File sub-issue on noetl/ops — Phase 1 deployment work (Helm chart + manifests + RBAC). Possibly also on noetl/noetl for the POOL_FILTER_MAP extension.
3. Build the system worker pool deployment on kind — image is existing ghcr.io/noetl/worker, no new code needed.
4. Write system/outbox_publisher.yaml — pure YAML using postgres + nats tool kinds. Lives in repos/ops/playbooks/system/ (or a new noetl/system-playbooks repo if it grows).
5. Register the playbook in the kind cluster catalog under system/outbox_publisher.
6. Validate end-to-end — insert a row into noetl.outbox (status=PENDING); confirm playbook claims it via NATS, publishes to NOETL_EVENTS, marks row PUBLISHED.
7. Retire the Python outbox publisher pod — flip the Deployment image or just scale to 0.

Related

• Umbrella: Rust Worker Migration — closed predecessor (worker side of migration; #30).
• Umbrella: Python Services to Rust — closed predecessor (compiled-Rust framing dropped; #45).
• Umbrella: Container Tool Callback — sibling, in-flight design (#43).
• Umbrella: Rust Worker Parity Gaps — sibling, tactical gaps (#47, #48).
• ADR: System Worker Pool and WASM Plug-in Surface (v2 in flight).
• noetl-server wiki: Runtime shape — implementation-level companion (will be revised to reflect playbook approach).
• noetl-ops wiki: System worker pool — deploy topology.