Umbrella WASM Plugin Compilation

Umbrella: Plug-in Compilation & Hot-Reload (WASM system playbooks)

Tracking issue: noetl/ai-meta#105 · Board: roadmap 3 · ADR: System Worker Pool and WASM Plug-in Surface

Goal

Let system-service logic be authored as playbooks on the system worker pool, with the option to compile that logic to a hot-reloadable compiled module managed as a replaceable plug-in library — the ADR's Phase 4 (WASM compilation), designed but not built. Its original home, #46, closed once the system pool shipped (Phases 1-2 live), so Phase 4 tracks here.

The shape (from the ADR — implement, don't re-litigate):

• wasmtime host inside the existing worker binary (no new binary), a dispatcher mode for WASM-flagged playbooks.
• Compile server-side at catalog-register; store module + digest in the catalog.
• Hot-reload via catalog version bump — workers cache by (path, version, digest), invalidate + reload on next claim, no restart.
• Capability-based imports via wasmtime's Linker (the wider system-pool capability set).
• The catalog is the managed, replaceable plug-in library.
• Interpreted execution stays the default + fallback; WASM is opt-in for plug-ins whose hot loop earns it. Per the ADR's batch-amortisation argument, projector / publisher / materialiser run fine interpreted today.

The open design decision (resolve first)

The ADR fixes where/when compilation happens but not the lowering model — how a playbook becomes a module:

• (A) Transpile playbook → WASM (automated; large — a lowering pass).
• (B) Hand-written Rust plug-ins → wasm32 against a plug-in ABI; the playbook is the catalog manifest (more tractable).
• (C) Hybrid — transpile common cases, hand-written escape hatch.

This choice sets the size of every round.

Decomposition

1. wasmtime host skeleton in the worker: load/invoke/drop a trivial .wasm; capability Linker with one host fn.
2. Catalog-version-keyed module cache + hot-reload.
3. Server-side compile-at-register + module/digest storage.
4. The lowering pass (per the decision) for a first real plug-in.
5. Port system/materialiser (± projector/publisher) to the compiled path; measure vs interpreted.

Recent activity

Date	What	Pointer
2026-06-18	(b) ops hardening — system-pool consumer name matches the KEDA scaler. Complements the server/worker execution_pool decline (server#232 + worker#114) with a committed-config fix: the system-pool deployment bound noetl_worker_system_rust while its KEDA scaler reads noetl_worker_pool_system — so the scaler watched a consumer the worker never created (no backlog scaling) and the live state had drifted to a hand-applied noetl_worker_system_kindtest. Aligned NATS_CONSUMER to noetl_worker_pool_system (scaler name + the noetl_worker_pool_<segment> convention). Single-stream consumer-filter model unchanged. Kind-validated (drive on, cursor+fan-out): the noetl_worker_pool_system consumer claimed all 44 orchestrate drives (Δ+44 = dispatched=applied=44); the shared consumer got only the 42 real steps; COMPLETED, 0 __orchestrate__ rows in noetl.event. Cluster restored (drive off). #108 stays open for (c) the default-flip.	ops#191
2026-06-17	Hot-replace pillar proven LIVE on running pods. The host's ensure_loaded_by_ref resolves the plug-in digest from the registry on every dispatch — so republishing the same path@version with new bytes hot-reloads the pool with no restart. Added the same-version-republish unit test (existing test only covered a version bump). Live kind proof: registered @1=variant A → ran → object at .../1.feather; republished @1=variant B (new digest, different key) → re-ran → object at .../HOTRELOAD-B.feather, all 8 worker restartCount still 0. The full runtime — load + run + capability-flush + hot-replace — is live-proven. Remaining (executor: author sugar; compiled materialiser port) deferred — the materialiser port needs nats_drain/events_project capabilities + the Arrow-in-wasm decision, owned by #104/#103.	worker#112
2026-06-17	Runtime complete — real data flows end to end. Live e2e exposed an empty object payload: wasm_config_to_ref read config.input but the server canonicalizes a step's input: to args. Fix reads args (input fallback). Re-validated on kind — input: {hello: world} now lands {"hello":"world"} (17 bytes) in noetl.object_store (was 0 bytes). The whole WASM dispatch path (server accept → tool_kind: "wasm" → worker host → digest → run → flush) carries real data. Runtime done; only the optional playbook→WASM lowering remains.	worker#110 (v5.31.1)
2026-06-17	Round 5 — WASM-flag convention + object-store (Feather) tier. Dispatch convention (ADR): executor: wasm + plugin {path,version} + capabilities → lowers to tool_kind: "wasm"; worker resolves digest → run_and_apply. Object-store endpoint: noetl.object_store + PUT/GET /api/internal/objects/{*key} (server#212); worker object_put repointed to it (worker#103). The flush is now boundary-correct + durable end to end.	ADR docs#181 · server#212 · worker#103
2026-06-16	Round 5 (dispatcher) — load, run, collect, flush. BufferingCapabilities (records event_publish/result_put/object_put as CapIntents — sync record, async flush, the local-first bridge) + WasmDispatcher (run = ensure_loaded + invoke + collect; run_and_apply = + flush). apply_intents flushes to the control plane: result/object → put_result (server-mediated; object base64-wrapped, interim until a Feather-tier endpoint), event → emit_event. Tested end to end with the real reference plug-in + a mock control plane. 23 plugin tests.	worker#100 · PR worker#101
2026-06-16	Round 5 (start) — reference Rust→wasm plug-in. plugins/reference-materializer: hand-written Rust → wasm32-unknown-unknown (303 bytes, no_std, no WASI, imports only noetl.object_put); host test loads the compiled .wasm and asserts the capability call — a real compiled plug-in (not WAT) runs end to end. Validates the full stack: registry → HTTP source → host → real plug-in. 18 plugin tests.	worker#96 · PR worker#97
2026-06-16	Round 4b (worker) — HTTP PluginSource. PluginSource made async; HttpPluginSource → GET /api/internal/plugins/{path}?version=&digest= (200→bytes, 404→NotLoaded, 409→stale-digest). Closes the loop: server registry → worker host → wasmtime. 17 plugin tests.	worker#94 · PR worker#95
2026-06-16	Round 04 (server) — plug-in module registry. noetl.plugin_module table keyed by (path,version) (digest + media_type + BYTEA), POST/GET /api/internal/plugins/{*path} — the durable backing for the worker's PluginSource. Mirrors result_store; 666 tests unaffected. v3.11.0.	server#209 · PR server#210
2026-06-16	Round 03 — materialiser capability ring + catalog loading. HostCapabilities (noetl.event_publish/result_put/object_put) registered on the Linker, reading key+payload from guest linear memory, dispatched to an injected impl (host does the real write → boundary holds); deny-by-default NullCapabilities; invoke_bytes_with capability injection; PluginSource/ensure_loaded catalog-keyed load (fetch-on-miss, cache-on-hit). 14 plugin tests. Server-side half (HTTP catalog source + compile-at-register) deferred to Round 4.	PR worker#93
2026-06-16	Round 02 — Arrow byte data-plane ABI. invoke_bytes: alloc-export + linear-memory hand-off so Arrow IPC/Feather buffers cross the boundary with no JSON serialization; a real Arrow IPC buffer round-trips byte-identical. Design recorded: compile target wasm32-wasip1/p2 but capabilities are NoETL host functions (not raw WASI fs/net); Arrow Flight for cross-network; OCI + runwasi distribution. 10 plugin tests.	PR worker#93 · ADR docs#181
2026-06-16	Round 01 — wasmtime host skeleton. WasmPluginHost: engine + capability Linker, module cache keyed by PluginKey{path,version,digest}, run invoke, hot-reload via evict_other_versions, capability ring by construction, REFERENCE_PLUGIN_WAT. Off-by-default wasm-plugin feature; 6 tests; not yet wired. Lowering = hybrid (confirmed). worker v5.23.0.	worker#92 · PR worker#93
2026-06-16	Umbrella opened — Phase 4 of the system-pool ADR re-homed here after #46 closed; #104 blueprint reshaped to route services to the plug-in ring	#105 · ADR Phase 4

Next concrete steps

1. Decide the lowering model — hybrid (C) chosen: reference plug-in first, then a playbook→WASM lowering pass.
2. Round 1: the wasmtime host skeleton — done (worker#93).
3. Round 2: Arrow byte data-plane ABI — done (worker#93).
4. Round 3: capability ring + catalog-keyed loading — done (worker#93). Worker host contract fixed; capability ring + ensure_loaded in place.
5. Round 4 (server): plug-in module registry — done (server#210). Table + register/serve endpoints; the live PluginSource backend.
6. Round 4b (worker): HTTP PluginSource — done (worker#95). Loop closed: server registry → worker host → wasmtime.
7. Round 5 (in progress):
   • ✅ reference Rust→wasm plug-in runs on the host (worker#97).
   • ✅ dispatcher — load from catalog + run + collect intents + flush to the control plane (worker#101). End to end: catalog → host → run → collect → flush.
   • ✅ WASM-flag dispatch convention designed (ADR docs#181): executor: wasm → tool_kind: "wasm" routing.
   • ✅ Feather-tier object-store endpoint (server#212)
     • worker object_put repointed to it (worker#103).
   • Routing — complete + live-validated ✅:
     • ✅ digest resolution (worker#105) · ✅ dispatch branch (worker#107) · ✅ feature-default flip (worker#108) · ✅ ToolKind::Wasm (server#214).
     • ✅ Full e2e on kind: a tool: {kind: wasm, plugin: {...}} playbook completed; the command routed to the host; the plug-in's object_put landed in noetl.object_store. PFT flip-safety also green.
   • Next: the worker input→args field fix (real payload flow); then the playbook→WASM lowering pass; port system/materialiser to the compiled path.
   • Then the playbook→WASM lowering pass; port system/materialiser to the compiled path; measure vs interpreted.

Related

• Umbrella: Event WAL Storage (#104) — the materialiser is a plug-in-ring system playbook; this umbrella is its compiled-hot-path capability.
• #46 (closed) — System Pool Design; this is its unbuilt Phase 4.