Document each operator by jachris · Pull Request #286 · tenzir/docs

jachris · 2026-04-29T11:38:22Z

Summary

Add reference pages for the new each and group operators.
Add a routing guide for dynamic fan-out with subpipelines, covering when to use each, group, fork, parallel, and load_balance.
Add reciprocal See Also links and update the operator index and sidebar.

_{⚙️ Code PR: tenzir/tenzir#5981}

github-actions · 2026-04-29T11:40:03Z

📦 Preview · ~~View →~~ · ⚪ Removed

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d1ca47295d

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Adds reference page for the new `each` operator, which spawns a subpipeline per incoming event with the event bound to `$this`. Adds reciprocal See Also links from `cron`, `every`, and `parallel`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

- Reword the source/output description (no more "or sinks them itself"). - Move the queueing note next to the parallel parameter. - Say the subpipeline must start with a source. - Drop the trivial "re-emit" example. - Replace the load_http example with a per-event sink example. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The previous example used `subscribe` and `to`, neither of which is ported to the new executor yet. Replace with `from`/`to_file` using `write_ndjson` inside the sink so the example actually runs under `tenzir --neo`. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Add a routing guide that explains how to use each and group for dynamic subpipeline fan-out. The guide compares them with fixed fan-out operators and shows per-event lookups and keyed sinks. Also add a group reference page and expand the each reference with a more realistic lookup example and guide links. Assisted-by: ChatGPT (pi)

mavam

Looks good. The new guide gives and the right guide-level treatment, and the reference pages now link readers to the broader fan-out guidance.

mavam

Looks good. The new guide gives each and group the right guide-level treatment, and the reference pages now link readers to the broader fan-out guidance.

Adds `each { ... }` which, for each incoming event, substitutes the event's record as `$this` into a fresh copy of the subpipeline and spawns it as a void-input (source) pipeline. Outputs from all subpipelines are forwarded. An optional `parallel=N` argument (default 10) caps concurrent instances; excess rows queue and drain one-by-one as subpipelines finish. This required adding `OperatorState::blocked` to `async.hpp` and updating the executor to stop consuming upstream messages when an operator returns `blocked`, while still processing subpipeline completions and control messages. Start with `executor.cpp` for the backpressure change, then `each.cpp` for the operator itself. The `state()` / `finish_sub` / `todo_` interaction is the key flow to follow. <sub> 📚 Docs PR: tenzir/docs#286 </sub>

github-actions Bot added the reference Reference documentation label Apr 29, 2026

jachris mentioned this pull request Apr 29, 2026

Add each operator tenzir/tenzir#5981

Merged

jachris commented Apr 29, 2026

View reviewed changes