Skip to content

feat: add batch consume to the source Hopper and adapters#135

Merged
joshua-temple merged 2 commits into
mainfrom
feat/source-batch
Jun 4, 2026
Merged

feat: add batch consume to the source Hopper and adapters#135
joshua-temple merged 2 commits into
mainfrom
feat/source-batch

Conversation

@joshua-temple

Copy link
Copy Markdown
Collaborator

What this change does

Adds batch consume to the source Hopper and adapters (one of the four roadmap items from #131). A BatchHandler (func(ctx, []Message) []Result) plus a WithBatch(size, maxWait) option lets a consumer process many messages per call instead of one.

The engine accumulates up to size messages per ordered lane, or until maxWait elapses on an injected clock, invokes the BatchHandler once, then settles each message by its positionally-aligned Result. Per-key ordering and the MaxInFlight bound are preserved; a result-count mismatch terms the unmatched messages as poison (ErrBatchResultCount), and a decode failure is isolated to its own message.

When a subscription advertises the Batched capability the engine hands it whole fetched batches and regroups by key; otherwise it falls back to per-message accumulation. Kafka exposes its polled records and JetStream its pull fetch through Batched. Batch is opt-in — per-message stays the default.

Why

Sinks, stores, and downstream APIs are often far more efficient per call when fed in batches, but doing that by hand breaks per-key ordering and the in-flight bound. Folding batching into the Hopper keeps those invariants intact while letting throughput-sensitive consumers opt in with one option.

Relates to: follows up the source Tier-1 PR (#131).

Checklist

  • Commits are signed off (git commit -s) per the DCO
  • Conventional commit messages (type: subject)
  • Tests added (table-driven memsource tests, Example, and a fuzz test asserting no loss/dup/reorder within a key); core at 92.1% coverage
  • Public API documented (godoc + Example + concurrency docs page)
  • mage check runs in CI across the workspace and the GOWORK=off source destinations

Add a BatchHandler (func(ctx, []Message) []Result) and a WithBatch(size,
maxWait) Hopper option, driven by RunBatch/ReceiveBatch. The engine
accumulates up to size messages per ordered lane, or until maxWait elapses
on an injected clock, invokes the BatchHandler once, then settles each
message by its positionally-aligned Result. Per-key ordering and the
MaxInFlight bound are preserved, a result-count mismatch terminates the
unmatched messages as poison (ErrBatchResultCount), and a decode failure is
isolated to its own message.

When a subscription advertises the Batched capability the engine hands it
whole fetched batches and regroups them by key, falling back to per-message
accumulation otherwise. Kafka exposes its polled records and JetStream its
pull fetch through Batched; batch is opt-in, per-message stays the default.

Covered by table-driven memsource tests (size/maxWait/drain triggers,
per-message result mapping, ordering, capability path), an Example, and a
fuzz test asserting batch + per-key ordering never loses, dups, or reorders
within a key. Documents a Batch consume section in the concurrency guide.

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
@joshua-temple joshua-temple merged commit dc32523 into main Jun 4, 2026
117 checks passed
@joshua-temple joshua-temple deleted the feat/source-batch branch June 4, 2026 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant