wiki: new noetl-events page + executor + sidebar + home cross-links (EE-4)
Catch-up wiki update for the EE-4 sequence that already landed on
the code side (cli#49, cli#50, server#38).
- New top-level page `noetl-events.md` documenting the workspace
crate published as 0.1.0 on 2026-06-05: purpose, public API
(`ExecutorEvent`, `EventSink`, `EventEmitter`, `NoopSink`), wire-
format alignment with the Python `EventEmitRequest`, the re-export
contract from `noetl-executor::events`, the path+version dep
recipe, release history.
- `_Sidebar.md` Architecture section now lists both crates.
- `executor-crate-architecture.md` workspace table adds the
noetl-events row; source-layout tree shows the `events/` folder
+ flags `executor/src/events.rs` as a 1-line re-export since 0.4.0.
- `Home.md` Distribution row mentions noetl-events on crates.io;
new Architecture section pointing at both crate pages.
Tracks the EE-4 sequence under noetl/ai-meta#49 +
noetl/cli#51 (R1 sub-issue) + noetl/cli#52 (R2 sub-issue).
docs(executor-arch): add GCS sink object_store semantic-divergence rows (noetl/ai-meta#31)
Adds two rows to the semantic-divergence table covering the migration of
SinkTarget::Gcs from gsutil subprocess to object_store crate in noetl/cli#48:
- GCS upload transport change (gsutil → GoogleCloudStorageBuilder)
- GCS credential resolution change (gsutil gcloud store → from_env() ADC chain)
Also marks the GCS sink → object_store deferred item as complete and updates
the R-1.1 "next phases" note.
Refs noetl/ai-meta#31
wiki: noetl-arrow-flight-client 0.5.0 (Phase C2.4 mTLS identity)
wiki: noetl-arrow-flight-client 0.4.0 (Phase C2.3 bearer token)
wiki: noetl-arrow-flight-client 0.3.0 (Phase C2.2 TLS client)
wiki: FlightResolver::connect example uses http:// (tonic scheme)
The `grpc://...` URL in the example doesn't actually work — tonic's
`Endpoint::from_shared` requires `http://` or `https://`; the
`grpc://` scheme other Flight clients (Java, Python pyarrow) accept
surfaces as `Bad :scheme header` at first request time.
Matches the corrected `derive_flight_endpoint` in noetl-tools 2.12.0
and the doc comment fix in noetl/cli#44.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs(architecture): note get_flight_info (Phase C1) + result_fetch playbook surface
Updates noetl-arrow-flight-client after Phase C1 landed
(noetl/noetl#644 + noetl/cli#43, crates.io 0.2.0):
- Workspace table row notes both releases (0.1.0 + 0.2.0) and
links to the matching server PR.
- "What it gives you" block now includes the get_flight_info
recipe with the FlightInfoSummary field rundown.
- New boundary-discipline paragraph cross-links to the
`result_fetch` tool kind in noetl-tools as the playbook-policy
layer that consumes this crate.
- Phase scope updated: A ✅, B ✅ (crates.io 0.1.0), C1 ✅
(crates.io 0.2.0), playbook surface ✅, C2 + multi-endpoint
sharding deferred.
docs(architecture): document noetl-arrow-flight-client workspace crate (R-2.3 Phase B)
Adds the new crate to the workspace-members table, extends the
source-layout block, and adds a dedicated `## noetl-arrow-flight-client`
section covering:
- What it gives you (FlightResolver + resolve/resolve_rows API).
- The Python server it pairs with (noetl/noetl#643, Phase A).
- Boundary discipline (thin RPC client; scrub stays server-side).
- Phase scope (A done, B done in noetl/cli#41, worker callsite
deferred, Phase C planned).
- Cross-links to the wiki page on noetl's Server API table
(arrow_flight_result_fetch).
executor-crate-architecture: add noetl-arrow-cache (R-2.1)
PR noetl/cli#39 added the noetl-arrow-cache workspace crate —
Rust mirror of Python's ArrowIpcSharedMemoryCache for same-node
zero-copy IPC. Cut as 0.1.0 alongside noetl 4.2.0.
Rewrite the page intro from a single-crate ("executor crate")
framing to a workspace ("noetl-cli workspace crates") framing,
add a workspace-member table, expand the source layout to
include arrow-cache/, and document the arrow-cache public
surface + cross-stack contract + data-plane role.
Refs noetl/ai-meta#30 — Appendix H Rust migration umbrella.
executor-crate-architecture: EE series complete (4 of 4)
After noetl/worker#11 merged (noetl-worker 3.0.0), the four-PR
cross-stack event envelope reconciliation series is done. All
four producers / consumers (executor, Rust server, Python
broker, worker) now emit + accept the same wire format on
/api/events.
Update the Deferred work bullet to mark PR-EE-3 ✅ with the
noetl-worker version + PR link.
Refs noetl/ai-meta#30.
executor-crate-architecture: EE-2 + EE-4 landed, only EE-3 remains
Update the Deferred work bullet to reflect the cross-stack
event envelope reconciliation series progress after the
noetl/server (Rust) + noetl/noetl (Python) PRs merged.
- PR-EE-2 ✅ noetl/server#6 + pipeline-fix #7 (noetl-server 2.0.1)
— EventRequest renamed name → event_type with alias; payload
accepts context alias; added optional event_id / status /
created_at.
- PR-EE-4 ✅ noetl/noetl#639 (noetl 3.0.0) — EventEmitRequest
accepts name / step / payload as validation aliases;
worker_id lifted to top-level; EventType Literal → str (real
finding: Literal was already dead code, production uses
dot-notation event types throughout).
- PR-EE-3 ⏳ noetl/worker — last step. Both servers now accept
either form so the worker switch can land any time.
Cross-link to noetl/server wiki event-envelope page.
docs(executor): record PR-EE-1 (event envelope reconciliation begins) + tidy PR-2d-1 publish notes
- Adds the PR-EE-1 row to the sub-PR landing-history table:
ExecutorEvent enriched with event_id / worker_id / meta optional
fields + payload serde alias, 0.3.0 -> 0.3.1, bin auto-bumped
to noetl 4.1.0.
- Removes the obsolete "pending publish" notes from the PR-2d-1
row now that 0.3.0 + noetl 4.0.0 are confirmed on crates.io.
- Rewrites the Event envelope reconciliation deferred-work bullet
with concrete progress: PR-EE-1 landed; PR-EE-2 (server Rust)
+ PR-EE-4 (Python) + PR-EE-3 (worker switch) remain. Worker
switches last when both servers accept the new shape.
Refs noetl/ai-meta#30 -- Appendix H umbrella.
Refs noetl/cli#37 -- the PR this row documents.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs(executor): add PR-2d-2 row to worker adoption history
Records the final R-1.2 worker-side sub-PR (noetl/worker#6).
The Worker adoption history table now covers:
- PR-2c (#2): condition surface; auto-bumped to 1.1.0, eventually
reaching crates.io as 1.1.2 after pipeline fix.
- PR-2d-2 (#6): CommandSource adoption; auto-bumped to 2.0.0 via
feat!: prefix (major bump).
Also fixes the PR-2c row's misleading "publish pending" note --
the worker release pipeline has been self-healing since
noetl/worker#5 fixed the actions: write permission.
Refs noetl/ai-meta#30 -- Appendix H umbrella.
Refs noetl/worker#6 -- the PR this row documents.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs(executor): record PR-2d-1 — CommandSource trait redesign + design decisions
- Adds the PR-2d-1 row to the noetl/cli sub-PR landing-history
table. Documents the breaking trait redesign (ack lifecycle +
4-state ClaimOutcome + AckHandle associated type + enriched
Command with render_context + attempts), the 0.3.0 bump, and
the auto-cascading noetl 4.0.0 release.
- Adds a dedicated "Design decisions" sub-section memorializing
the four load-bearing API choices (Pulled<H> as struct not
tuple; AckHandle as associated type not generic; error carrier
as String not anyhow::Error; render_context + attempts
#[serde(default)] for forward compat) plus the lifecycle
invariant ("exactly one ack/nack per pulled handle, sequential
per task"). Each row gives the rationale so future readers
understand why these choices land where they do.
- Updates the deferred-work section to mark the noetl-worker
release pipeline gap as fixed (worker#4 + #5 — pipeline now
self-healing as of noetl-worker 1.1.2).
Refs noetl/ai-meta#30 -- Appendix H umbrella.
Refs noetl/cli#35 -- the PR this row documents.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs(executor): R-1.2 publishing + worker adoption history
Extends the executor-crate-architecture wiki page beyond R-1.1
complete with two new sections:
1. **R-1.2 sub-PR landing history** on noetl/cli:
- PR-1 (#32): publish noetl-executor 0.1.0 to crates.io;
fixes the broken release pipeline that had been failing for
3 consecutive runs (bin stuck at 2.17.1 vs 2.24.0).
- PR-2a (#33): align execution_id to i64 across the crate;
semver-major to 0.2.0; bin auto-bumped to noetl 3.0.0 since
semantic-release picked up the breaking-change language.
- PR-2b (#34): add structured Condition + 12-variant Operator
+ evaluate_structured_condition for worker adoption; minor
bump to 0.2.1; bin auto-bumped to noetl 3.1.0.
2. **Worker adoption history** on noetl/worker:
- PR-2c (noetl/worker#2): adopts noetl-executor = "0.2" dep;
case_evaluator.rs 437 -> 344 LoC; condition primitives
delegate to the executor; pull-loop control flow stays in
the worker per § H.10. Tagged v1.1.0 but crates.io publish
pending (worker's semantic-release.yml is missing the
follow-up `gh workflow run release.yml` step).
3. **Deferred work** section: event envelope reconciliation
(worker WorkerEvent ↔ server EventEmitRequest ↔ executor
ExecutorEvent), GCS sink → object_store (R-3 scope), and the
worker release pipeline gap (same bug noetl/cli had
pre-PR-1).
Refs noetl/ai-meta#30 -- Appendix H umbrella.
Refs noetl/cli#32, noetl/cli#33, noetl/cli#34, noetl/worker#2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs(executor): R-1.1 complete -- PR-2d row + final stats
Adds the PR-2d row to the sub-PR landing-history table. Wrap-up
PR: end-to-end integration tests (12 new in `executor/tests/`),
crate README, lib.rs module-list update. Closes noetl/cli#19.
The "After PR-2c-8, the remaining R-1.1 work is PR-2d" paragraph
becomes "R-1.1 is complete after PR-2d ships" with pointers to
R-1.2/R-1.3 (worker adoption), R-2 (Arrow), R-3 (object_store
including the GCS sink follow-up).
Stats footer rewritten as a final tally:
- playbook_runner.rs: -1,082 net cumulative
- noetl-executor: 80 unit + 12 integration = 92 tests
- Workspace: 174 passing (was 162)
- Tool kinds wired: 4 / inline-with-helpers: 3 / unsupported: 1
Refs noetl/cli#19 -- closing it via PR #31.
Refs noetl/cli#31 -- the PR this row documents.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs(executor): record PR-2c-8 -- Tool::Auth + Tool::Sink helpers
Adds the PR-2c-8 row to the sub-PR landing-history table. Like
PR-2c-7, this row documents a sub-PR shaped as "codify the
decision + extract pure helpers" rather than "migrate the code":
- Tool::Auth context-update side-effects extracted into
auth_context_updates (Vec<(key, value)>) so callers apply the
set_variable calls explicitly. Preserves the CLI's pre-PR-2c-8
variable-set ordering.
- Tool::Sink format conversion extracted into format_sink_payload
+ json_to_csv. Sink targets (file/duckdb/gcs) stay inline per
§ H.10 since noetl-tools has no file/cloud-storage equivalents.
- Both dispatch arms now bail loudly with helper-pointing
messages.
No semantic-divergence rows added — `format_sink_payload` is
byte-for-byte equivalent to the inline impl; `json_to_csv` lifted
verbatim; `auth_context_updates` preserves the original ordering.
Stats footer bumped:
- playbook_runner.rs: 1,636 -> 1,606 (-1,082 net cumulative)
- noetl-executor unit tests: 65 -> 80
- workspace tests: 147 -> 162
- new code in noetl-executor: ~3,050 -> ~3,400 LoC
Refs noetl/cli#19 -- R-1.1 sub-issue.
Refs noetl/cli#30 -- the PR this row documents.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs(executor): record PR-2c-7 -- § H.10 codification for Tool::Playbook
Adds the PR-2c-7 row to the sub-PR landing-history table. Unlike
the previous five tool-replacement rows, this one DOESN'T migrate
an inline implementation -- it encodes the § H.10 architectural
finding that `Tool::Playbook` is not bridgeable: sub-playbook
recursion is part of the CLI's tree walker by design. The bridge
dispatch arm bails loudly with a § H.10-citing message instead of
silently returning an empty outcome; the variable-preparation
step (DSL v2 input / DSL v1 args merging) moves into a new
`prepare_sub_playbook_vars` helper that's reusable and testable.
No semantic-divergence rows added for this PR — no execution path
changed. This is the first sub-PR in the series where the right
shape is "codify the decision," not "migrate the code."
Stats footer bumped:
- playbook_runner.rs: 1,638 -> 1,636 (-1,052 net cumulative)
- noetl-executor unit tests: 57 -> 65 (+8 for prepare_sub_playbook_vars + bail test)
- workspace tests: 139 -> 147
- new code in noetl-executor: ~2,800 -> ~3,050 LoC
Refs noetl/cli#19 -- R-1.1 sub-issue.
Refs noetl/cli#29 -- the PR this row documents.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs(executor): record PR-2c-6 -- Tool::DuckDb via noetl-tools bridge
Adds the PR-2c-6 row to the sub-PR landing-history table and four
semantic-divergence rows under [#28]:
- duckdb SELECT result envelope: noetl-tools' {columns, rows,
row_count} reshapes back to CLI's rows-array shape
- duckdb non-SELECT result envelope: {affected_rows: N} maps back
to the CLI's literal {"status": "ok"} string
- duckdb params: silently ignored -> bound as JSON at `?`
placeholders (feature gain)
- duckdb path resolution + mkdir: CLI keeps owning these at the
call site (bridge has no knowledge of the playbook directory)
Stats footer bumped:
- playbook_runner.rs: 1,641 -> 1,638 (-1,050 net cumulative)
- noetl-executor unit tests: 48 -> 57
- workspace tests: 130 -> 139
- new code in noetl-executor: ~2,550 -> ~2,800 LoC
Refs noetl/cli#19 -- R-1.1 sub-issue.
Refs noetl/cli#28 -- the PR this row documents.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs(executor): record PR-2c-5 -- Tool::Http via noetl-tools bridge
Adds the PR-2c-5 row to the sub-PR landing-history table and four
semantic-divergence rows under [#27]:
- http transport: curl subprocess -> reqwest direct
- http auth (GCP ADC): gcloud shellout -> gcp_auth crate
- http JSON body: verbatim -> auto-detected (reqwest sets
Content-Type)
- http result envelope: pass-through reshape so 4xx/5xx come back
inside the {status, body} envelope (not as anyhow errors)
Plus a fifth row noting that Tool::Auth now shares the same GCP
ADC code path via the new resolve_auth_to_bearer helper.
Stats footer bumped:
- playbook_runner.rs: 1,656 -> 1,641 (-1,047 net cumulative)
- noetl-executor unit tests: 39 -> 48
- workspace tests: 80 -> 130 (was undercounting one binary)
- new code in noetl-executor: ~2,250 -> ~2,550 LoC
Refs noetl/cli#19 -- R-1.1 sub-issue.
Refs noetl/cli#27 -- the PR this row documents.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
docs(executor): add PR-2c-4 row + shell semantic-divergence entries + bump stats
- Adds PR-2c-4 (Tool::Shell wired) to sub-PR landing-history table.
- New rows in 'Documented semantic divergences' table for the shell
stdout streaming loss (real-time line-by-line -> end-of-execution
burst) and the result-envelope unwrap.
- Stats footer bumped:
- playbook_runner.rs: 1,727 -> 1,656
- noetl-executor LoC: ~2,100 -> ~2,250
- noetl-executor unit tests: 34 -> 39
- Workspace tests: 75 -> 80
Refs noetl/cli#19
Refs noetl/cli#26
docs(executor): add PR-2c-3 row + semantic-divergence table + bump stats
- Adds PR-2c-3 (Tool::Rhai wired) to sub-PR landing-history table.
- New 'Documented semantic divergences' table cross-references the
PR for each behaviour delta: timestamp() shape change, HTTP
curl->reqwest, get_gcp_token gcloud->gcp_auth. Subsequent
per-tool replacements (PR-2c-4 shell, PR-2c-5 http, etc.) add
rows to this table when they cross behaviour lines.
- Stats footer bumped:
- playbook_runner.rs: 1,964 -> 1,727
- noetl-executor LoC: ~1,850 -> ~2,100
- noetl-executor unit tests: 29 -> 34
- Workspace tests: 75 (34 noetl-executor + 41 noetl bin)
Refs noetl/cli#19
Refs noetl/cli#25
docs(executor): add PR-2c-2 to sub-PR landing history + bump stats
Adds PR-2c-2 row (tools_bridge adapters) to the sub-PR table and
updates the stats footer:
- noetl-executor LoC: ~1,400 → ~1,850
- noetl-executor unit tests: 17 → 29
- Workspace tests: 41 (29 noetl-executor + 12 noetl bin)
Refs noetl/cli#19
docs: add noetl-executor workspace crate architecture page
Documents the workspace-member crate introduced in v2.17.1+ via R-1.1
of the Rust migration roadmap (Appendix H of the global hybrid cloud
blueprint).
The page covers:
- Source layout (executor/ as a workspace member of repos/cli)
- What lives in noetl-executor (playbook types, template + condition
+ capabilities utilities, events shape, tools_bridge scaffold,
worker-only CommandSource trait) and why
- What does NOT live in noetl-executor (CLI tree walker, CLI inline
tool implementations, worker NATS pull loop, RunOutcome envelope)
- The § H.10 architectural finding (tree walker vs pull-model) that
rescoped noetl-executor from a control-loop crate into a
utilities-and-types crate during R-1.1 PR-2b
- Cross-references to Appendix H, Polars endpoint, Arrow data plane,
tracking issues
- Sub-PR landing history: PR-1 (#20), PR-2a (#21), PR-2b (#22),
PR-2c-1 (#23 open)
- Stats: playbook_runner.rs 2,688 -> 1,964 lines; noetl-executor
~1,400 LoC; unit tests 0 -> 17
Sidebar gains a new "Architecture" section linking the page.
Closes the wiki-maintenance Rule 1 gap that opened during PR-1
("first touch of substantive surface ships with a wiki page")
and continued through PR-2a / PR-2b / PR-2c-1. Each future
sub-PR in this arc (PR-2c-2 onwards) extends the sub-PR landing
history table and any module-level additions.
Refs noetl/cli#19
Refs noetl/ai-meta#30