What happened?
Beam 2.75.0
SolaceIO write transform has two "writer" types - batched (sends 50 messages per batch) and streaming (sends one message at a time). With both of those writers, there is the issue when using the writer in a batch-mode Beam pipeline.
The issue is that the batch pipeline will frequently end its run before Solace publish results are output by the SolaceIO write transform.
In case of UnboundedBatchedSolaceWriter, the issue is that it relies on a timer to "finalize" outputing the publish results. In a batch pipeline, timers are not really triggering as expected, at least the pipeline will not wait the 10 seconds which is configured in the code for the timer to fire. Instead, the pipeline will just end - output is lost.
In case of UnboundedStreamingSolaceWriter, there is no timer but it calls publishResults from the @FinishBundle method which assumes that by that time there are Solace publish results to output. When there aren't, the batch pipeline again will just end, as there is nothing holding back the watermark and it does not care that there are some Solace ACK callbacks left waiting. Output is lost.
CC @stankiewicz @iht
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components
What happened?
Beam 2.75.0
SolaceIO write transform has two "writer" types - batched (sends 50 messages per batch) and streaming (sends one message at a time). With both of those writers, there is the issue when using the writer in a batch-mode Beam pipeline.
The issue is that the batch pipeline will frequently end its run before Solace publish results are output by the SolaceIO write transform.
In case of
UnboundedBatchedSolaceWriter, the issue is that it relies on a timer to "finalize" outputing the publish results. In a batch pipeline, timers are not really triggering as expected, at least the pipeline will not wait the 10 seconds which is configured in the code for the timer to fire. Instead, the pipeline will just end - output is lost.In case of
UnboundedStreamingSolaceWriter, there is no timer but it callspublishResultsfrom the@FinishBundlemethod which assumes that by that time there are Solace publish results to output. When there aren't, the batch pipeline again will just end, as there is nothing holding back the watermark and it does not care that there are some Solace ACK callbacks left waiting. Output is lost.CC @stankiewicz @iht
Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
Issue Components