Umbrella WASM Plugin Compilation

Umbrella: Plug-in Compilation & Hot-Reload (WASM system playbooks)

Tracking issue: noetl/ai-meta#105 · Board: roadmap 3 · ADR: System Worker Pool and WASM Plug-in Surface

Goal

Let system-service logic be authored as playbooks on the system worker pool, with the option to compile that logic to a hot-reloadable compiled module managed as a replaceable plug-in library — the ADR's Phase 4 (WASM compilation), designed but not built. Its original home, #46, closed once the system pool shipped (Phases 1-2 live), so Phase 4 tracks here.

The shape (from the ADR — implement, don't re-litigate):

• wasmtime host inside the existing worker binary (no new binary), a dispatcher mode for WASM-flagged playbooks.
• Compile server-side at catalog-register; store module + digest in the catalog.
• Hot-reload via catalog version bump — workers cache by (path, version, digest), invalidate + reload on next claim, no restart.
• Capability-based imports via wasmtime's Linker (the wider system-pool capability set).
• The catalog is the managed, replaceable plug-in library.
• Interpreted execution stays the default + fallback; WASM is opt-in for plug-ins whose hot loop earns it. Per the ADR's batch-amortisation argument, projector / publisher / materialiser run fine interpreted today.

The open design decision (resolve first)

The ADR fixes where/when compilation happens but not the lowering model — how a playbook becomes a module:

• (A) Transpile playbook → WASM (automated; large — a lowering pass).
• (B) Hand-written Rust plug-ins → wasm32 against a plug-in ABI; the playbook is the catalog manifest (more tractable).
• (C) Hybrid — transpile common cases, hand-written escape hatch.

This choice sets the size of every round.

Decomposition

1. wasmtime host skeleton in the worker: load/invoke/drop a trivial .wasm; capability Linker with one host fn.
2. Catalog-version-keyed module cache + hot-reload.
3. Server-side compile-at-register + module/digest storage.
4. The lowering pass (per the decision) for a first real plug-in.
5. Port system/materialiser (± projector/publisher) to the compiled path; measure vs interpreted.

Recent activity

Date	What	Pointer
2026-06-16	Round 01 — wasmtime host skeleton. WasmPluginHost: engine + capability Linker, module cache keyed by PluginKey{path,version,digest}, run invoke, hot-reload via evict_other_versions, capability ring by construction, REFERENCE_PLUGIN_WAT. Off-by-default wasm-plugin feature; 6 tests; not yet wired. Lowering = hybrid (confirmed). worker v5.23.0.	worker#92 · PR worker#93
2026-06-16	Umbrella opened — Phase 4 of the system-pool ADR re-homed here after #46 closed; #104 blueprint reshaped to route services to the plug-in ring	#105 · ADR Phase 4

Next concrete steps

1. Decide the lowering model — hybrid (C) chosen: reference plug-in first, then a playbook→WASM lowering pass.
2. Round 1: the wasmtime host skeleton — done (worker#93).
3. Round 2: wire the version-keyed cache to the live catalog (load a module by catalog (path, version, digest)); widen the capability ring beyond noetl.emit (event publish, result-store write, object-store put).
4. Round 3: server-side compile-at-register + module/digest storage.
5. Round 4: the hybrid lowering pass for a first real plug-in; port system/materialiser to the compiled path; measure vs interpreted.

Related

• Umbrella: Event WAL Storage (#104) — the materialiser is a plug-in-ring system playbook; this umbrella is its compiled-hot-path capability.
• #46 (closed) — System Pool Design; this is its unbuilt Phase 4.