docs(deployment-spec): add Secret providers section (GCP + k8s env + RBAC) Documents NOETL_K8S_* (k8s secrets provider, server#96) and backfills the previously-undocumented GCP Secret Manager env vars. Notes the server SA needs secrets:[get,list] RBAC. Tracks noetl/ai-meta#61.

kadyapam committed Jun 6, 2026

docs(event-envelope): terminal playbook.failed on orchestrator evaluate failure (v2.27.2) Deterministic evaluate failures (invalid template in a step body, unknown next-arc step) now emit a terminal playbook.failed instead of stranding the execution in RUNNING. server#95 (closes server#94), surfaced by ai-meta#54.

kadyapam committed Jun 6, 2026

docs(event-envelope): add EE-5 follow-up — lax decode of integer execution_id / event_id Closes the wire-type drift between the Rust worker (noetl-events::ExecutorEvent emits execution_id / event_id as JSON integers) and the server's strict- String decode (which 422'd with "invalid type: integer, expected a string"). Shipped via noetl/server#57 (v2.19.1) — two custom serde adapters (deserialize_string_or_i64 + Option<String> variant) applied to: - EventRequest.execution_id - EventRequest.event_id - BatchEventRequest.execution_id Outbound encoding stays String for browser JSON-number precision. 6 new unit tests pin the dual-shape contract. Tracks noetl/ai-meta#55 + noetl/server#56.

kadyapam committed Jun 5, 2026

design: codify Phase G keychain-backed shard endpoints (deferred from R4) Records the design decision made 2026-06-04: R4 ships with the flat `NOETL_SHARDS` env-var DSN model; the keychain-backed DB-resident shard model becomes Phase G work. Two reasons captured: 1. Cross-cloud heterogeneity (GCP workload identity vs AWS IAM vs static password) needs a connection record with auth_kind + credential_alias, not a flat URL. 2. `execution-model.md` § "Secrets and credentials rule" lists tenant DB DSNs as business-logic credentials; same applies to shard DSNs at the cross-cloud boundary. Phase G shape sketched: `noetl.shard_endpoint` table on the cluster master with credential_alias references into `noetl.keychain`; same model extended to `noetl.result_ref` so external storage pointers carry their own keychain alias. R4 continues unchanged: env-var DSN through R4-5 + kind validation. R4-1's env-var path documented as kind-dev fallback in the long-term layering.

kadyapam committed Jun 4, 2026

wiki: GET /api/runtime/shard-info diagnostic endpoint (Phase F R3b-1) Per wiki-maintenance.md Rule 2a, route + env docs land in lockstep with the code (noetl/server#46 + the v2.12.0 PR). - Sharding section gains a new "Diagnostic endpoint (Phase F R3b-1)" subsection with the route shape, response example, validation rules, and companion-endpoint context (gateway twin + integration test in ops). - Network surface row lists the new route alongside the existing /api/runtime/* paths. Refs noetl/ai-meta#49 Refs noetl/server#46

kadyapam committed Jun 4, 2026

wiki: deployment-spec gains NOETL_SHARD_INDEX / NOETL_SHARD_COUNT (Phase F R2) Per wiki-maintenance.md Rule 2a, env var docs land in the same change set as the code (noetl/server#44 + the upcoming PR for v2.11.0). - NOETL_* table gains both vars with the *why* behind each. - New "## Sharding" section explains the default no-op behavior, enablement procedure for the Phase F R4+ cutover, and the routing key derivation (hash(execution_id) % N via twox-hash with fixed seed 0). Refs noetl/ai-meta#49 Refs noetl/server#44

kadyapam committed Jun 4, 2026

wiki: deployment-specification page (Phase F R1.5 follow-up) Codifies the deployment shape for noetl-server: runtime contract, network surface, resources, health probes, FULL env-var catalogue with the why behind each one, secrets handling, observability hooks, kind-validation procedure. Captures the NOETL_SERVER_MACHINE_ID variable added in Phase F R1.5 (noetl/server#42) alongside the existing env surface that wasn't previously documented (NOETL_HOST, NOETL_PORT, NOETL_DEBUG, NOETL_ENCRYPTION_KEY, NOETL_INTERNAL_API_TOKEN, NOETL_PUBLIC_SERVER_URL, NOETL_DISABLE_METRICS, NOETL_AUTO_RECREATE_RUNTIME, NOETL_RUNTIME_SWEEP_INTERVAL, NOETL_RUNTIME_OFFLINE_SECONDS, NOETL_SCHEMA; POSTGRES_*; NATS_*; HOSTNAME / COMPUTERNAME / RUST_LOG). Sidebar gains an "Operations" section linking the new page; Home references it in the Pages list. Going forward this page is the single source of truth for env vars + ports + dependencies; any code change that touches std::env::var or envy::from_env must update it in the same change set per the new agents/rules/wiki-maintenance.md Rule 2a (landing separately on ai-meta). Refs noetl/ai-meta#49 Refs noetl/server#42

kadyapam committed Jun 4, 2026

wiki: Phase F R1 — sharding design doc (noetl/server#40) R1 of Phase F under noetl/ai-meta#49. Codifies the survey findings into a durable design doc the next rounds (R2 server- side shard_id; R3 gateway-side dispatch; R4 DB sharding; R5 cutover) refer to. New page `sharding-design.md` covering: - TL;DR with the 5 design decisions. - Why the architecture is already ready (NATS subjects already execution-aware; orchestrator is stateless re: execution; advisory locks are command-scoped; transient variables already keyed by execution_id). - Shard assignment: `hash(execution_id) % N` on the full i64, with rationale for rejecting `timestamp % N` (time hotspots) and `machine_id % N` (clusters by generating machine). - Routing strategy: gateway-aware for F; server-aware proxy as Phase G fallback if needed. - Per-execution table partition list (5 tables) with citations. - Cluster-wide table list (4 tables) + sync strategy (single- master for F; per-shard read replicas as G). - Full endpoint inventory: per-execution (route by execution_id) vs cluster-wide (any shard answers); the awkward GET /api/executions list case called out. - Load-bearing prerequisite: app-side snowflake IDs. Per observability.md Principle 3, execution_id should be generated in the application, not by the DB. Must land before R4 — otherwise per-shard DB sequences cluster IDs to one shard. Migration shape laid out as R1.5. - Phase F decomposition table (R1–R5). - Open question NOT decided in R1: DB partition scheme (Citus vs per-shard schemas vs per-shard DBs) — R4 decides. - Out-of-scope for F: multi-region, per-tenant sharding, dynamic re-sharding. Sidebar + Home cross-linked. Closes noetl/server#40 Refs noetl/ai-meta#49

kadyapam committed Jun 4, 2026

wiki: refresh event-envelope page for noetl-events crate adoption (EE-4) Catch-up wiki update for noetl/server#38 (EE-4 PR 3) — server v2.9.0 takes a direct dep on the canonical noetl-events crate and anchors the shared subset of EventRequest to noetl_events::ExecutorEvent via bidirectional conversion impls + four wire-compat tests. - "Current shape" section bumps to v2.9.0+, adds inline comments on which fields are server-only (result_kind, result_uri, event_ids, actionable, informative), points at the new adoption section for the design call. - New "noetl-events crate adoption (EE-4 follow-up, noetl/ai-meta#49)" section: the three-PR sequence (cli#49, cli#50, server#38), the reason the two types stay distinct (server's EventRequest carries 5 legitimately server-only fields + String wire format for execution_id / event_id for browser JSON-number precision), the four wire-compat tests pinning the shared-subset round-trip semantics, the conversion impl shapes. - "Cross-stack envelope state" table now lists noetl-events as the canonical row + flags noetl-executor 0.4.0+ as a re-export + notes noetl-server v2.9.0+ anchors via From/TryFrom impls. - "See also" section adds links to the noetl-events cli wiki page, the crates.io entry, and the noetl/ai-meta#49 umbrella. Tracks noetl/server#39 (sub-issue closed by server#38).

kadyapam committed Jun 4, 2026

docs(routes): Phase A complete — ui_schema landed, /api/runtimes noted Phase A of noetl/ai-meta#49 wraps with the ui_schema port in v2.2.0. Updates the implementation-status callout: every Phase A endpoint is now wired and reports byte-identical JSON via the parity harness. Lists the two notable design calls (ui_schema port, /api/runtimes removal) and the Phases B–F still ahead. Refs noetl/ai-meta#49

kadyapam committed Jun 3, 2026

docs(routes): add implementation-status callout for Phase A #49 - Adds /api/worker/pools to the wired-routes table - Notes /api/catalog/{path}/ui_schema as pending (noetl/server#18) - Notes /api/runtimes as removed in Phase A round 4 (noetl/server#19) Refs noetl/ai-meta#49

kadyapam committed Jun 3, 2026

add Runtime shape page (compiled core + plug-in ring design proposal) Implementation-level companion to the docs-site ADR (https://noetl.dev/docs/architecture/system_pool_and_wasm_plugins). Documents the proposed shape of the Rust replacement for the Python control plane: - One crate, one image, four binaries (server / publisher / projector / system_pool) selected via ``--mode=`` flag. - Shared library inventory (config, db pool, nats, envelope, scrub, metrics, tracing, snowflake) used by all four. - Per-binary main loop sketches and metrics list. - Capability surface for system WASM modules. - Helm deployment shape — one image, multiple Deployments with different ``args:``. - Sequencing: publisher first, then projector, then server, then system_pool. Wired into Home + _Sidebar. Tracked under: - noetl/ai-meta#45 — compiled rewrite. - noetl/ai-meta#46 — system pool plug-in surface.

kadyapam committed Jun 2, 2026

event-envelope: EE-4 truly complete after noetl#641 + kind validation The EE umbrella's "all four producers/consumers accept the same wire format" promise had a latent gap until 2026-05-31: EE-4 (noetl/noetl#639) added the EventEmitRequest schema with aliases but it lived in dead code — the actually-mounted /api/events endpoint at core/events.py used the legacy EventRequest model with no aliases. noetl/noetl#641 (noetl 4.0.0) closes the gap by adding the same validation_alias declarations to core/models.py::EventRequest + execution_id int→str coercion. Kind validation against the live broker confirmed every EE-3 wire assertion holds. Add a "EE-4 finalisation post-merge" section documenting the gap + the fix + linking to the kind-validation summary on noetl/ai-meta#30. Refs noetl/ai-meta#30.

kadyapam committed May 31, 2026

event-envelope: EE series complete after PR-EE-3 landed noetl/worker#11 (noetl-worker 3.0.0) replaced WorkerEvent with ExecutorEvent re-export. Updates the cross-stack reconciliation table to ✅ for all four PRs and replaces the "pre-EE-3" worker row with the post-EE-3 shape (top-level worker_id; context replaces payload; event_id None today with the server default firing pending the application-side snowflake follow-up). Refs noetl/ai-meta#30.

kadyapam committed May 31, 2026

event-envelope: reflect EE-1 + EE-2 + EE-4 cross-stack reconciliation After PR-EE-4 landed in Python (noetl#639) and noetl-server's PR-EE-2 (#6) + pipeline-fix #7 (auto-republished as v2.0.1) the event envelope is now aligned across executor / Rust server / Python server — only EE-3 (worker switch) remains. - Document EventRequest's new optional fields (event_id, status, created_at) per observability.md Principle 3 (app-side snowflakes) — server falls back to its existing defaults when the caller omits them, so pre-EE clients keep working. - Add cross-stack reconciliation progress table (EE-1/2/4 done, EE-3 last). - Add full envelope state table showing all four shapes accept the same wire format. - Document the dot-notation discovery that surfaced while landing EE-4: the Python EventType Literal was already dead code (real production events use step.exit, call.done, command.completed, etc. — never the underscored variants).

kadyapam committed May 31, 2026

docs(wiki): scaffold server wiki (Home + event-envelope + sidebar) First substantive content for the noetl-server wiki. Replaces the "Welcome to the server wiki!" stub with three pages. - **Home.md**: documents noetl-server as the Rust port of the control plane (twin to the Python implementation in noetl/noetl/server during migration). Architecture diagram (event flow: worker -> /api/events -> PostgreSQL -> NATS -> worker claims). Full module layout table covering config / db / error / handlers (events, commands, catalog, credentials, executions, variables, keychain, runtime, dashboard) / engine (commands, evaluator, orchestrator, state) / nats / playbook / crypto. Migration plan: Python -> Rust over R-1.3 -> R-2. - **event-envelope.md**: the four-shape divergence currently in flight (Rust server EventRequest, Python EventEmitRequest, worker WorkerEvent, executor ExecutorEvent). Reconciliation roadmap (v2 endpoint additive, then worker/CLI switch, then v1 deprecation). Explains why ExecutorEvent's shape is the right target. Cross-references the noetl/ai-meta#30 umbrella issue for tracking. - **_Sidebar.md**: navigation; cross-links to cli + worker + tools + noetl + ops wikis. All cross-references use noetl/<repo>#<NN> format per the ai-meta cross-repo issue-linking discipline. Refs noetl/ai-meta#30 -- Appendix H umbrella. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

kadyapam committed May 30, 2026

Initial Home page

kadyapam committed May 30, 2026