You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
ref(eap-outcomes): updating committing logic, more metrics (#7813)
## Summary
This PR improves the committing and backpressure handling in the
accepted outcomes consumer, and adds a `--commit-frequency-sec`
parameter to control how often commit requests are emitted.
### Changes
- **`CommitOutcomes`**: Added a configurable `commit_frequency` (default
10s) so that offsets are only committed on a timer rather than on every
poll. Offsets are now only tracked after a successful produce — if the
produce step rejects a message, offsets are **not** advanced. On `join`,
a forced commit is issued regardless of the timer.
- **`OutcomesAggregator`**: Fixed backpressure handling. Previously,
when the next step rejected a message during flush, the batch was
silently restored and the flush effectively became a no-op. Now, a
rejected message is carried over and retried on the next `poll`. While a
message is carried over, incoming `submit` calls return
`MessageRejected` so upstream applies backpressure. The `join` path also
retries carried-over messages until they succeed or the deadline
elapses.
- **`ProduceOutcome`**: Added a `timer!` metric
(`accepted_outcomes.batch_produce_ms`) to track how long each batch
produce takes.
- **`accepted_outcomes_consumer` CLI**: Added `--commit-frequency-sec`
(default `10`) that is plumbed through to `CommitOutcomes`.
### Future considerations
- Might have this just run `RunTask` instead of `RunTaskInThreads` given
that producing should be pretty quick and batching will be what takes
longer, might be better to prevent flushing a batch until the first once
is done producing. If we have multiple threads and something fails that
would be more data loss if we are committing early
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>