Umbrella Python Services To Rust

Umbrella — Python Services to Rust (compiled rewrite) — CLOSED 2026-06-02

ai-task: noetl/ai-meta#45 (CLOSED) · Opened: 2026-06-02 · Closed: 2026-06-02 (same day) · Status: ❌ Superseded by Umbrella: System Pool Design · ADR (v2 revised): System Worker Pool and WASM Plug-in Surface

Closed 2026-06-02 before any code work landed. The "compiled Rust rewrite of publisher + projector + server" framing was dropped within hours of being filed: the team decided that publisher + projector belong as system playbooks (not compiled Rust binaries), executed on the system worker pool. The HTTP server stays Python for now; its migration is a separate concern from the playbook-plug-in story. See the closing comment on the issue for the full audit trail. All future migration work flows through Umbrella: System Pool Design.

Goal

Replace the three remaining Python pods in the active runtime path with Rust equivalents:

Pod (today)	Module	Function
noetl-server	Python FastAPI / uvicorn	Catalog, /api/execute, /api/events, SSE
noetl-outbox-publisher	python -m noetl.outbox_publisher	Postgres event_outbox → NATS NOETL_EVENTS
noetl-projector-0	python -m noetl.projector	NATS noetl.events.> → Postgres noetl.event

The noetl-worker Python pool (still claims agent tool kind) is tracked under Umbrella: Rust Worker Migration not here.

Architecture decision

After the placement analysis on #45, the chosen shape:

One crate (noetl/server), one image (ghcr.io/noetl/server:<v>), four binaries (server / publisher / projector / system_pool) selected via --mode= flag. Postgres-style: one source tree, multiple roles via separate binaries that share the same library.

See noetl/server wiki — Runtime shape for the implementation-level layout, and noetl/ops wiki — System worker pool for the deploy shape.

Sequencing

Smallest surface first to amortise the shared-library scaffolding:

Step	Binary	Replaces	Status
1	--mode=publisher	noetl-outbox-publisher	Not started
2	--mode=projector	noetl-projector-0	Not started
3	--mode=server	noetl-server	Not started
4	--mode=system	(new — see system-pool umbrella)	Not started

Steps 1 + 2 drop two of the three Python pods even before step 3 ships — useful intermediate checkpoint.

Step 4 is the system worker pool (per Umbrella: System Pool Design); depends on step 3 for catalog reads.

Per-step acceptance

Step 1 — --mode=publisher

• Reads from Postgres outbox table via LISTEN/NOTIFY (latency win over today's poll loop).
• Publishes to NOETL_EVENTS NATS stream with the same envelope contract Python uses today.
• Same metrics surface (noetl_publisher_*) per agents/rules/observability.md.
• Kind validation: repos/ops/automation/development/noetl.yaml reapplied with the publisher Deployment image flipped to ghcr.io/noetl/server:<v> --mode=publisher. Smoke test: insert into event_outbox, observe NATS message land within 100ms.
• Wiki update: noetl-server wiki Home table + Runtime shape page; noetl-ops wiki manifests list.

Step 2 — --mode=projector

• Sharded StatefulSet preserved — pod name → consumer name, filter subject hashes by execution_id.
• Batch-INSERT into noetl.event with ON CONFLICT (event_id) DO NOTHING for idempotency.
• Same metrics (noetl_projector_*).
• Kind validation: same playbook; smoke test: publish to noetl.events.<subject>, observe row in noetl.event table.

Step 3 — --mode=server

The big one. Route inventory in noetl-server wiki — full HTTP API parity with the Python implementation:

• /api/catalog/list, /api/catalog/register, /api/catalog/resource, /api/catalog/{path}/ui_schema
• /api/execute, /api/executions/{id}, /api/executions/{id}/events
• /api/events (the put_result boundary the Rust worker already uses)
• /api/credentials/*, /api/keychain/*
• /api/runtime/contract (the gateway + SPA contract)
• SSE endpoints

Kind validation: full regression suite passes against the new server.

Step 4 — --mode=system

See Umbrella: System Pool Design for the design + tracking.

Recent activity

Date	Event
2026-06-02	Issue filed during session 2026-06-02; placement analysis comment captured the same-crate/four-binary decision.
2026-06-02	ADR merged via noetl/docs#176 → published at https://noetl.dev/docs/architecture/system_pool_and_wasm_plugins. Bumped via ai-meta@eda6eed.
2026-06-02	Cross-linked wiki pages went live: noetl-server wiki — Runtime shape and noetl-ops wiki — System worker pool.
2026-06-02	No code work started yet.

Next concrete steps

1. Start with --mode=publisher. Smallest LoC, narrowest surface, validates the shared-library scaffolding. Estimated ~1 week of dedicated work.
2. Add the shared library in step 1. Don't try to bring it to perfection before step 2 — adjust as the second binary surfaces new requirements.
3. Sequence the deployment cutover — publisher first, then projector, then server. Each step independently rollback-able by reverting the Deployment image.

Related

• Umbrella: System Pool Design — sibling design, depends on this.
• Umbrella: Rust Worker Migration — covers the worker-side migration (this umbrella covers non-worker).
• ADR: System Worker Pool and WASM Plug-in Surface.
• noetl-server wiki: Runtime shape.
• noetl-ops wiki: System worker pool.