docs(orchestrator): cursor/claim loop mode (mode: cursor)
wiki: deployment-specification page (Phase F R1.5 follow-up)
Codifies the deployment shape for noetl-server: runtime contract,
network surface, resources, health probes, FULL env-var catalogue
with the why behind each one, secrets handling, observability hooks,
kind-validation procedure.
Captures the NOETL_SERVER_MACHINE_ID variable added in Phase F R1.5
(noetl/server#42) alongside the existing env surface that wasn't
previously documented (NOETL_HOST, NOETL_PORT, NOETL_DEBUG,
NOETL_ENCRYPTION_KEY, NOETL_INTERNAL_API_TOKEN,
NOETL_PUBLIC_SERVER_URL, NOETL_DISABLE_METRICS,
NOETL_AUTO_RECREATE_RUNTIME, NOETL_RUNTIME_SWEEP_INTERVAL,
NOETL_RUNTIME_OFFLINE_SECONDS, NOETL_SCHEMA; POSTGRES_*; NATS_*;
HOSTNAME / COMPUTERNAME / RUST_LOG).
Sidebar gains an "Operations" section linking the new page; Home
references it in the Pages list.
Going forward this page is the single source of truth for env vars
+ ports + dependencies; any code change that touches std::env::var
or envy::from_env must update it in the same change set per
the new agents/rules/wiki-maintenance.md Rule 2a (landing
separately on ai-meta).
Refs noetl/ai-meta#49
Refs noetl/server#42
wiki: Phase F R1 — sharding design doc (noetl/server#40)
R1 of Phase F under noetl/ai-meta#49. Codifies the survey
findings into a durable design doc the next rounds (R2 server-
side shard_id; R3 gateway-side dispatch; R4 DB sharding;
R5 cutover) refer to.
New page `sharding-design.md` covering:
- TL;DR with the 5 design decisions.
- Why the architecture is already ready (NATS subjects already
execution-aware; orchestrator is stateless re: execution;
advisory locks are command-scoped; transient variables already
keyed by execution_id).
- Shard assignment: `hash(execution_id) % N` on the full i64,
with rationale for rejecting `timestamp % N` (time hotspots)
and `machine_id % N` (clusters by generating machine).
- Routing strategy: gateway-aware for F; server-aware proxy as
Phase G fallback if needed.
- Per-execution table partition list (5 tables) with citations.
- Cluster-wide table list (4 tables) + sync strategy (single-
master for F; per-shard read replicas as G).
- Full endpoint inventory: per-execution (route by execution_id)
vs cluster-wide (any shard answers); the awkward
GET /api/executions list case called out.
- Load-bearing prerequisite: app-side snowflake IDs. Per
observability.md Principle 3, execution_id should be generated
in the application, not by the DB. Must land before R4 —
otherwise per-shard DB sequences cluster IDs to one shard.
Migration shape laid out as R1.5.
- Phase F decomposition table (R1–R5).
- Open question NOT decided in R1: DB partition scheme (Citus vs
per-shard schemas vs per-shard DBs) — R4 decides.
- Out-of-scope for F: multi-region, per-tenant sharding, dynamic
re-sharding.
Sidebar + Home cross-linked.
Closes noetl/server#40
Refs noetl/ai-meta#49
add Runtime shape page (compiled core + plug-in ring design proposal)
Implementation-level companion to the docs-site ADR
(https://noetl.dev/docs/architecture/system_pool_and_wasm_plugins).
Documents the proposed shape of the Rust replacement for the
Python control plane:
- One crate, one image, four binaries (server / publisher /
projector / system_pool) selected via ``--mode=`` flag.
- Shared library inventory (config, db pool, nats, envelope,
scrub, metrics, tracing, snowflake) used by all four.
- Per-binary main loop sketches and metrics list.
- Capability surface for system WASM modules.
- Helm deployment shape — one image, multiple Deployments with
different ``args:``.
- Sequencing: publisher first, then projector, then server, then
system_pool.
Wired into Home + _Sidebar.
Tracked under:
- noetl/ai-meta#45 — compiled rewrite.
- noetl/ai-meta#46 — system pool plug-in surface.
docs(wiki): scaffold server wiki (Home + event-envelope + sidebar)
First substantive content for the noetl-server wiki. Replaces
the "Welcome to the server wiki!" stub with three pages.
- **Home.md**: documents noetl-server as the Rust port of the
control plane (twin to the Python implementation in
noetl/noetl/server during migration). Architecture diagram
(event flow: worker -> /api/events -> PostgreSQL -> NATS ->
worker claims). Full module layout table covering config /
db / error / handlers (events, commands, catalog, credentials,
executions, variables, keychain, runtime, dashboard) /
engine (commands, evaluator, orchestrator, state) / nats /
playbook / crypto. Migration plan: Python -> Rust over
R-1.3 -> R-2.
- **event-envelope.md**: the four-shape divergence currently in
flight (Rust server EventRequest, Python EventEmitRequest,
worker WorkerEvent, executor ExecutorEvent). Reconciliation
roadmap (v2 endpoint additive, then worker/CLI switch, then
v1 deprecation). Explains why ExecutorEvent's shape is the
right target. Cross-references the noetl/ai-meta#30
umbrella issue for tracking.
- **_Sidebar.md**: navigation; cross-links to cli + worker +
tools + noetl + ops wikis.
All cross-references use noetl/<repo>#<NN> format per the
ai-meta cross-repo issue-linking discipline.
Refs noetl/ai-meta#30 -- Appendix H umbrella.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>