--maxBundleTimeMs is accepted but has no effect. A bundle is closed when it reaches --maxBundleSize elements or when a watermark arrives, so on a sparse stream the elements already fed to a bundle are not emitted until the next watermark, and the watermark cannot advance past them.
The natural implementation — closing the bundle from a wall-clock punctuator — produces duplicate output against a real broker. With it enabled, an integration test running two chained GroupByKeys across four partitions emits its single group six times, reproducibly, and the count keeps climbing after the input has stopped.
What has been ruled out:
- Metrics folding. Splitting the same input across many bundles does not change a user counter (
MetricsAcrossBundlesTest).
ProcessorContext.commit(). The duplication happens with the commit request removed, and does not happen with the punctuator disabled but the commit still requested.
So it appears to be closing a Fn-API bundle from a punctuator rather than from record processing. Note that Kafka Streams sets commitOffsetNeeded inside StreamTask#process(), so data produced from a punctuator sits outside that accounting; KAFKA-6906 was a bug of that shape, though it was fixed long before the version in use here.
Part of #18479.
--maxBundleTimeMsis accepted but has no effect. A bundle is closed when it reaches--maxBundleSizeelements or when a watermark arrives, so on a sparse stream the elements already fed to a bundle are not emitted until the next watermark, and the watermark cannot advance past them.The natural implementation — closing the bundle from a wall-clock punctuator — produces duplicate output against a real broker. With it enabled, an integration test running two chained GroupByKeys across four partitions emits its single group six times, reproducibly, and the count keeps climbing after the input has stopped.
What has been ruled out:
MetricsAcrossBundlesTest).ProcessorContext.commit(). The duplication happens with the commit request removed, and does not happen with the punctuator disabled but the commit still requested.So it appears to be closing a Fn-API bundle from a punctuator rather than from record processing. Note that Kafka Streams sets
commitOffsetNeededinsideStreamTask#process(), so data produced from a punctuator sits outside that accounting; KAFKA-6906 was a bug of that shape, though it was fixed long before the version in use here.Part of #18479.