dashboard: #111 e2e off-server orchestrate topology coverage + server-API-only gap
- Home: add #111 Active-umbrella row; lead the Last-refreshed line with the
e2e topology rig + the server-API-only gap assessment + the two operator
decisions.
- Sessions-Log: prepend the 2026-06-18 #111 entry.
- Umbrella-System-Pool-Design: Recent-activity row + Last-update header for the
committed off-server e2e rig + the surfaced decisions.
wiki(#108 c): drive default-flip CLOSED — Home + Sessions + Releases + umbrella
The worker-driven orchestrator drive defaults ON (server v3.28.0, server#233);
#108 closed. Home: #108 → Recently closed, #107 row updated, ecosystem-map
server v3.28.0 / worker v5.33.0, Last refreshed. Sessions-Log + Releases
entries; Umbrella-System-Pool-Design recent-activity row.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
wiki: Phase 2.b blocked on #52 (js_consume tool op)
wiki: Phase 2.a.2 path-based pool routing shipped
- Umbrella-System-Pool-Design status matrix: Phase 2.a.2 ⏳ → ✅
(PRs noetl/noetl#661 v4.11.0 + noetl/ops#144 merged; kind-
validated via consumer-delivery deltas).
- Sessions-Log: new top-of-page entry covering the routing design
(path wins over kind), the implementation surface (4 emit call
sites + LRU cache), kind validation results, and the small
outbox-playbook auth-block follow-up before end-to-end works.
- Releases: timeline row for noetl/noetl v4.11.0 + ops @ 66282e7,
plus a v4.11.0 row in the per-repo noetl/noetl table.
- Home: active-umbrella row reflects Phase 2.a complete; remaining
blockers (playbook auth fix → Phase 2.b projector → Phase 3
more playbooks) called out.
wiki: Phase 2.a.3 system pool env wiring (noetl/ops#143 merged)
- Umbrella-System-Pool-Design: mark Phase 2.a.3 ✅; PR ref updated
from "in PR / #143 open" to "shipped / #143 merged; manifest
applied to kind".
- Sessions-Log: new top-of-page entry "Phase 2.a.3 keychain env
wiring shipped" documenting the merge, the manifest patch
(NOETL_KEYCHAIN_ENV_VARS), the new pod's verified env, and the
remaining Phase 2.a.2 sub-task.
- Releases: new timeline row for noetl/ops @ b6dd205.
- Home: active-umbrella row updated — Phase 2.a.3 now ✅, Phase
2.a.2 server-side routing called out as the last remaining
sub-task before end-to-end kind validation.
wiki: refresh diagrams to reflect current implementation state
After Phase 1.a + 1.b + 2.a.1 shipped, the diagrams update to show
actual state vs the original "design-only" sketches:
- Home — Today's topology diagram refreshed:
- noetl-server box shows both Python + Rust versions with ✅ on
/api/internal/* completion
- System worker pool added to the deploy map with NOETL_INTERNAL_
API_TOKEN annotation
- outbox-publisher + projector marked "↓ to-be-retired"
- "What's live on kind right now" callout below the diagram
- Umbrella-System-Pool-Design — two new diagrams + one refresh:
- NEW "Implementation status (2026-06-02)" ASCII matrix showing
Phase 1.a / 1.b / 2.a.1 ✅ and 2.a.2 / 2.a.3 / 2.b / 3 / 4 ⏳
- NEW "Auth flow — Secret to API call" diagram showing the
five-hop pipeline from K8s Secret through env mount,
keychain-env-allowlist, AuthResolver, HTTPS, server gate
- Existing system playbook execution flow diagram unchanged
(the contract didn't shift)
- Data-Access-Boundary — adds the auth flow diagram (same five-
hop pipeline) with explicit "configuration-only, no Rust code"
callout below. This is the page operators land on when
wondering "how does a system playbook get authenticated."
The repeated auth-flow diagram is intentional — Data-Access-
Boundary is the convention page (rule-of-thumb readers); Umbrella-
System-Pool-Design is the umbrella page (people picking up Phase
2 work). Both audiences benefit from the same diagram in their
landing context.
Per agents/rules/wiki-maintenance.md Rule 0a: diagrams update in
the same change set as the work that produces them. This commit
catches up the diagrams to the actual state after Phase 2.a.1
shipped.
wiki: Phase 2.a.1 system/outbox_publisher playbook shipped
After noetl/ops#142 merged + ai-meta@8bf66f1 pointer bump:
- Sessions-Log adds "2026-06-02 (late evening 3) — Phase 2.a.1
system/outbox_publisher playbook shipped" entry at top.
Captures the playbook contract, the auth flow diagram, and
the surprise that Phase 2.a.3 is configuration-only.
- Umbrella-System-Pool-Design Phase 2.a system/outbox_publisher
checkbox flips from pending to ✅ + adds notes about the auth
alias + the routing-extension dependency (Phase 2.a.2 still
pending before end-to-end validation possible).
- Home dashboard #46 status: "Phase 1.a ✅ + Phase 1.b ✅ +
Phase 2.a.1 ✅; Phase 2.a.2 server routing + 2.a.3 deployment
env (PR #143) remain".
- Releases Timeline adds noetl/ops@b77244e row.
Per agents/rules/wiki-maintenance.md Rule 0a: every session
lands its wiki updates in the same change set.
Refs noetl/ai-meta#46
Refs noetl/ai-meta#50
wiki: Phase 1.b shipped — system worker pool deployed on kind
After noetl/ops#141 (system pool deployment manifests) and
noetl/ops#139 (validation rig) merged + ai-meta@8f40fc7 pointer
bump:
- Sessions-Log adds "2026-06-02 (late evening 2) — Phase 1.b
system worker pool deployed" entry at top. Captures all four
manifests, the kind validation output, the open Phase 2.a
question, and the next-session plan.
- Umbrella-System-Pool-Design Phase 1.b checklist flips from
pending to ✅, with sub-items noting what's deferred to Phase
2.a (POOL_FILTER_MAP, server-side validation) and what's
further deferred (Helm values, drift-guard test entry).
- Home dashboard #46 status text updated: "Phase 1.a ✅
kind-validated (Python); Phase 1.b ✅ system pool deployed on
kind; Phase 2.a playbook is next".
- Releases Timeline adds the noetl/ops@a529e55 row.
Per agents/rules/wiki-maintenance.md Rule 0a: every session
lands its wiki updates in the same change set.
Refs noetl/ai-meta#46
wiki: Phase 1.a kind-validated on Python
After kind validation of noetl/noetl#659 / #660 (Python v4.10.1) and
noetl/server#12 / #13 (Rust v2.1.1):
- Sessions-Log adds "2026-06-02 (late evening) — Phase 1.a kind-
validated on Python" entry at top. Captures the three real-world
bugs found, the validation harness output, and DB-side
verification.
- Releases page Timeline + per-repo log gain v4.10.1 (Python fix
release) and v2.1.1 (Rust fix release) rows.
- Umbrella-Rust-Server-Port Phase C status: Python column flips to
"kind-validated"; Rust column stays "unit-tested" (kind validation
deferred to side-by-side deploy). Three-bug summary added.
- Umbrella-System-Pool-Design Phase 1.a same status flip + harness
link.
- Home dashboard status text updated:
- #49: "Phase C ✅ shipped + kind-validated on Python; Rust unit-
tested, kind-validation deferred"
- #46: "Phase 1.a ✅ kind-validated (Python); Phase 1.b + Phase 2
ready to start"
Per agents/rules/wiki-maintenance.md Rule 0a: every session lands
its wiki updates in the same change set. This bump satisfies that
for the kind validation work.
Refs noetl/ai-meta#46
Refs noetl/ai-meta#49
wiki: Phase C complete on both Python + Rust servers
After noetl/server#12 merged (mirroring noetl/noetl#659 in Rust):
- Sessions-Log adds "2026-06-02 (evening) — Phase C Rust side
shipped (both servers complete)" entry at top. Captures the
implementation details, the new deps (subtle, serial_test), the
test counts, and the phase-status flip.
- Releases page Timeline + per-repo log gain the noetl/server@fdf775a
row (no version tag yet; awaits semantic-release).
- Umbrella-Rust-Server-Port Phase C table — Rust column flips from
⏳ to ✅ for all 6 endpoints + auth gate + tracing spans.
- Umbrella-System-Pool-Design Phase 1.a — same flip with the
c94075e SHA reference for the server pointer bump.
- Home dashboard — both primary umbrellas' status texts updated:
- #49: "Phase C ✅ on both Python + Rust; other phases pending"
- #46: "Phase 1.a ✅ on both servers; Phase 1.b + Phase 2 pending
kind validation"
Per agents/rules/wiki-maintenance.md Rule 0a: every session that
lands meaningful cross-repo work updates the ai-meta wiki in the
same change set. This bump satisfies that for noetl/server#12.
Refs noetl/ai-meta#49
Refs noetl/ai-meta#46
wiki: Phase C Python side landed (noetl v4.10.0)
After noetl/noetl#659 merged + tagged as v4.10.0:
- Sessions-Log gets a new "2026-06-02 (evening) — Phase C Python
side shipped" entry at top with the endpoint inventory, the PR
link, and the next-session plan.
- Releases page Timeline gains the v4.10.0 row + a per-repo entry
under "noetl/noetl".
- Umbrella-Rust-Server-Port Phase C table flips Python column to
✅ (5 endpoints + auth gate); Rust column stays ⏳ pending the
mirror sub-issue.
- Umbrella-System-Pool-Design Phase 1.a same flip + ai-meta SHA
reference (cdb597d).
- Home dashboard active-umbrella table status text updated — #49
shows "Phase C Python ✅; Rust mirror pending"; #46 shows
"Phase 1.a Python ✅; Phase 1.b + Phase 2 pending kind
validation".
Per agents/rules/wiki-maintenance.md Rule 0a: every session that
lands meaningful cross-repo work updates the ai-meta wiki in the
same change set. This bump satisfies that for #659.
Refs noetl/ai-meta#49
Refs noetl/ai-meta#46
add sharding sequence + system playbook execution flow diagrams
Continuation of the 2026-06-02 diagram pass:
1. **Sharding sequence diagram** added to Phase F of #49
(Umbrella-Rust-Server-Port.md). Shows:
- Initial /api/execute call lands on any shard via the
gateway/ingress load balancer.
- Subsequent shard-aware calls (eg. GET /api/executions/{id})
route by shard_id = hash(execution_id) % N.
- Outbox claim distributes per-shard via system pool worker
pod ownership.
- Sharding rules table — what's shared (catalog, credentials)
vs what's sharded (execution, event, command, outbox).
- Migration path: N=1 first, then scale to N=3.
2. **System playbook execution flow** added to Phase 2 of #46
(Umbrella-System-Pool-Design.md, under system/outbox_publisher).
Shows:
- KEDA HTTP scaler triggers off pending-count endpoint.
- System pool worker claims command from NATS.
- Playbook executes 4 steps: claim_batch (HTTP) → publish_batch
(NATS iterator) → mark_published (HTTP) → mark_failed
(per-row failure branch).
- Worker emits call.done → outbox → next iteration picks it
up (bootstrap-tolerant cycle closure).
- Auth gate noted (system-pool-sa-token on /api/internal/*).
Together with the previously-added interlock + Python-trajectory
+ data-access-boundary diagrams, the dashboard now answers:
- What's deployed today
- How the two primary umbrellas connect
- Where we're heading (Python footprint trajectory)
- Why workers can't touch DB directly
- How sharding will work
- How a system playbook actually runs
All from wiki pages a developer can scan in a few minutes.
add architectural diagrams across dashboard pages
After session 2026-06-02 (late afternoon) request: "add that
diagrams too" — make the architecture readable at a glance from
any wiki page a developer might land on.
Four diagrams added:
1. **Two-umbrella interlock** (#49 + #46) — visual of how the
Rust server port's Phase C unblocks the system pool's Phase
2. Same API surface produced by one, consumed by the other.
Added to:
- Home.md (architecture-at-a-glance section)
- Umbrella-Rust-Server-Port.md
- Umbrella-System-Pool-Design.md
2. **Long-term Python trajectory** — three-column view:
today / after #46 Phase 2 / long-term. Shows Python
becoming a container payload, not a platform runtime.
Added to:
- Home.md
- Umbrella-System-Pool-Design.md
3. **Data access boundary** — workers + system worker pool
call server HTTP API; only the server talks to Postgres
directly; external-subsystem playbooks (auth) go direct to
external systems.
Added to:
- Data-Access-Boundary.md (convention page)
4. **Today's deploy topology** (pre-existing, kept) — gateway
→ server → outbox/projector + NATS → worker pools.
Already on Home.md.
Together these give a developer landing on Home immediately:
- What's deployed today
- The two primary in-flight umbrellas + how they interlock
- The Python footprint trajectory
Each Umbrella-* page carries its relevant subset so a developer
opening a specific umbrella also gets the visual context.
Issue comments mirror these diagrams:
- #49 — interlock diagram (gh issue comment 4605222980)
- #46 — interlock + trajectory diagrams (gh issue comment
4605227009)
add Data-Access-Boundary convention page + corrected playbook examples
After 2026-06-02 (afternoon) standing instruction: NoETL platform
data is accessible via server API only. Workers (including
system worker pool) call the API, never the DB directly.
Changes:
- Data-Access-Boundary.md — new convention page pointing to the
authoritative rule in agents/rules/. TL;DR + why (connection
pool, sharding, consistency) + auth exception + decision tree.
- _Sidebar.md — adds Data Access Boundary under Conventions.
- Home.md — adds Data Access Boundary to the conventions index
with link reference.
- Umbrella-System-Pool-Design.md:
- Adds "Data access boundary" section explaining the rule.
- Phase 1 split into 1.a (server-side API endpoints) and 1.b
(system worker pool deployment). 1.a lists the five new
/api/internal/* endpoints needed before any Phase 2
playbook can run.
- Phase 2 playbook examples corrected:
- system/outbox_publisher uses tool: http (claim/mark) +
tool: nats (publish).
- system/projector uses tool: nats (pull batch) + tool: http
(project).
dashboard pivot — system playbooks become primary migration path
After 2026-06-02 afternoon architecture decision: rest of Python →
Rust migration goes via compilable + pluggable system playbooks
on the system worker pool, not via more compiled binaries.
Home.md:
- Last-refreshed note updated with the pivot summary.
- Active-umbrella table trimmed from 6 → 4 (closed #30 and #45
moved into Recently Closed section).
- #46 marked PRIMARY MIGRATION UMBRELLA at the top of the table.
- Sessions log section adds the afternoon entry pointing to the
pivot.
Umbrella pages:
- Umbrella-Rust-Worker-Migration — marked CLOSED 2026-06-02 with
successor pointer to #46.
- Umbrella-Python-Services-To-Rust — marked CLOSED 2026-06-02
(same day; compiled-rewrite framing dropped within hours).
- Umbrella-System-Pool-Design — fully rewritten as the primary
migration umbrella. Captures expanded scope: Phase 1
(runtime), Phase 2 (publisher + projector playbooks), Phase 3
(additional system playbooks), Phase 4 (WASM compilation).
Next concrete steps listed.
Sessions-Log.md:
- Adds "2026-06-02 (afternoon) — Architecture pivot: system
playbooks" entry at the top with full context.
Closes the architectural ambiguity: there is now ONE primary
migration umbrella (#46), and it captures the platform-extends-
itself-via-playbooks design.
inject dates across dashboard pages for developer scanability
Per session 2026-06-02 request: "inject dates so other developers
can get a sense what is done and when and what is going on now."
Changes across pages:
- Home — adds Opened/Last update columns to active umbrella table,
appends Recently Closed (last 7 days) table with closure dates,
adds Last commit column to ecosystem map with PR links, expands
Sessions log + Releases sections with explicit dates.
- Umbrella-Rust-Worker-Migration — adds Opened/Last update at
top, dates each R-1/R-2/R-3 sub-task with completion date,
surfaces closed-issue links per phase, reshapes Recent activity
as a dated table.
- Umbrella-Rust-Worker-Parity-Gaps — adds Surfaced/Filed/Last
update at top, dates the regression run that surfaced them,
cites the master execution_id.
- Umbrella-Python-Services-To-Rust — adds Opened/Last update at
top, reshapes Recent activity as a dated table including the
ADR merge SHA.
- Umbrella-System-Pool-Design — same shape, captures the design
conversation timeline (issue file → matrix capture → ADR merge
→ wiki cross-links).
- Umbrella-Container-Tool-Callback — adds Opened/Last update,
dated activity table.
- Umbrella-Event-Envelope — adds Started/Last update, dated each
EE-1/EE-2/EE-3 landing, marked EE-4 as blocked on #45 step 1.
- Repo-Map — adds Last commit column to every repo table.
- Releases — adds Timeline section (all repos, last 14 days,
sorted by date).
Goal: a developer scanning any page can see "what's done, when,
what's next" without opening the issue tracker.
initialise NoETL ecosystem dashboard
Per session 2026-06-02 request: "use ai-meta wiki as todo umbrella
tracker for all issues and submodule repos to keep links and
description of what we do across all repos."
The ai-meta wiki becomes the single pane of glass for the NoETL
platform — cross-repo umbrellas, ecosystem map, release history,
session decisions, agent-coordination conventions.
Initial page set:
Dashboard:
- Home — active-umbrella table, ecosystem map, recent sessions,
release pointer, conventions index
- _Sidebar — navigation
Ecosystem reference:
- Repo Map — every submodule, role, current version, wiki link
- Releases — per-repo release log with GitHub Release URLs
- Sessions Log — chronological agent session entries
Active umbrellas (one page per ai-task issue):
- Umbrella-Rust-Worker-Migration (#30)
- Umbrella-Container-Tool-Callback (#43)
- Umbrella-Python-Services-To-Rust (#45)
- Umbrella-System-Pool-Design (#46)
- Umbrella-Rust-Worker-Parity-Gaps (#47 + #48)
- Umbrella-Event-Envelope (#51 EE-1..EE-4)
Conventions (pointers into agents/rules/):
- Issue-Tracking
- Wiki-Convention (the ai-meta vs per-repo split)
- Handoffs
- Deployment-Validation
- Execution-Model
- Observability
Cross-link discipline: this wiki uses bare slugs internally and
full URLs to per-repo wikis. Per-repo wikis link back to
specific pages here (not Home).