Umbrella Rust Worker Migration

Umbrella — Rust Worker Migration (Appendix H)

ai-task: noetl/ai-meta#30 · Opened: 2026-05-30 · Last update: 2026-06-02 · Status: In flight (R-2 / R-3 phases) · Blueprint: Appendix H of the global hybrid cloud architecture

Goal

Migrate the runtime hot path from Python to Rust while protecting the production Python platform. Phased rollout so each step is independently shippable + rollback-able.

Phases

R-0 — Blueprint update ✅ (2026-05-29)

Appendix H drafted in noetl/docs#174. Landed 2026-05-29.

R-1 — Shared noetl-executor crate ✅ (2026-05-27 → 2026-05-31)

CLI and worker depend on the same execution-engine crate; ≥80% shared code path.

Sub-task	Done	Status
R-1.1 PR-1 — crate skeleton	2026-05-27	✅
R-1.1 PR-2a — extract YAML types	2026-05-27	✅
R-1.1 PR-2b — extract parser + command-gen	2026-05-28	✅
R-1.1 PR-2c — replace inline tools with noetl-tools (8 sub-PRs)	2026-05-29 → 2026-05-30	✅ closes noetl/cli#19
R-1.1 PR-2d — integration tests + docs	2026-05-30	✅
R-1.2 PR-1 — publish executor 0.1.0 to crates.io	2026-05-30	✅
R-1.2 PR-2a — i64 execution_id alignment	2026-05-30	✅
R-1.2 PR-2b/c — worker condition module adopts executor	2026-05-30	✅
R-1.2 PR-2d — worker NatsCommandSource impl	2026-05-31	✅
R-1.2 PR-2e — worker observability harness	2026-05-31	✅ closes #32

R-2 — Arrow + Flight ✅ (2026-05-28 → 2026-06-01)

Sub-task	Done	Status
R-2.1 — arrow-rs in noetl-tools	2026-05-28	✅
R-2.2 PR-A/B — tabular encoder fallback	2026-05-29	✅
R-2.3 Phase A — pyarrow.flight.FlightServerBase on Python server	2026-05-30	✅
R-2.3 Phase B — noetl-arrow-flight-client crate	2026-05-31	✅
R-2.3 Phase C1 — get_flight_info discovery	2026-05-31	✅
R-2.3 Phase C2 — mTLS + token auth	2026-06-01	✅ closes #33

R-3 — Tool-kind parity 🔄 (2026-06-01 → in flight)

Rust worker reaches tool-kind parity for the production playbook mix. KEDA scales Rust and Python pools off the same NATS subjects (per-pool routing).

Sub-task	Date	Status	Tracking
R-3 Phase B-1 — nats tool kind in noetl-tools	2026-06-02	✅ closed	#38
R-3 Phase B-2 — mcp tool kind in noetl-tools	2026-06-02	✅ closed	#39
R-3 Phase B-3 — bump worker noetl-tools ^2.16	2026-06-02	✅ closed	#40
R-3 Phase B-4 — KEDA dual-scaling for Rust pool	2026-06-02	✅ closed	#41
R-3 Phase C-1 — agent tool kind routing decision	2026-06-02	✅ closed	#42
R-3 Phase C-2 — container tool kind callback design	2026-06-02 → in flight	🔄 design	#43
Per-pool NATS subject routing (PRs 1-6)	2026-05-31 → 2026-06-01	✅	#42
KEDA multi-trigger scaler	2026-06-01	✅	noetl/noetl#657
Multi-arch image publishing (worker v5.9.0)	2026-06-02	✅ closed	#44
Routing flag flip on prod-shaped env	2026-06-01	✅	noetl/ops#138
task_sequence tool kind	2026-06-02 surfaced	⏳ Ready to pick up	#47
Credential alias resolution	2026-06-02 surfaced	⏳ Ready to pick up	#48

R-4 — _handle_event_inner decision point

Six-month review based on production load data — port to Rust if handle_event p90 is materially constraining, otherwise declare hybrid the durable design. Not yet reached.

Recent activity

Date	Event
2026-06-02	Kind validation against worker v5.9.0 surfaced #47 + #48 (regression gaps). Routing scheme proven end-to-end. Session details.
2026-06-02	Worker v5.9.0 republished as native multi-arch via noetl/worker#39 (replaces failed QEMU build).
2026-06-02	R-3 Phase B/C-1 issues all closed: #38-#42 (5 closures).
2026-06-01	Ops PR-5 routing flag flip merged via noetl/ops#138. Server publishes start landing on noetl.commands.shared.<eid> / noetl.commands.python.<eid>.
2026-05-31	Rust worker observability harness PR-2e merged; /metrics endpoint + snowflake event_id.
2026-05-30	PR-2c noetl-tools registry sweep complete (8 sub-PRs); noetl/cli#19 closed.
2026-05-29	Appendix H blueprint landed via noetl/docs#174.

Next concrete steps

1. Pick up #47 (task_sequence tool kind). Read Python implementation in repos/noetl/noetl/worker/process.py first to decide between Option A (implement in noetl-tools) and Option B (route to Python pool).
2. Pick up #48 (credential alias resolution). Add ControlPlaneClient::get_credential if not present; wire into tool dispatch path so auth: "<alias>" resolves before reaching the tool.
3. Continue #43 container tool kind callback design.
4. Re-run kind regression after #47 + #48 land; target ≥45/53 COMPLETED (today's 17/53 + the ~35 gap fixes).

Related

• Umbrella: Rust Worker Parity Gaps — #47 + #48 grouped.
• Umbrella: Container Tool Callback — #43.
• Umbrella: Python Services to Rust — #45, the non-worker services parallel umbrella.
• Releases — worker, tools, cli release history.