fix: data-plane-http - HttpSink loosing messages #3417 #3543
Conversation
…uce a multiplicity of parts with given binary content.
…ransfer of multiple message under various partition size settings. Demonstrates a bug in the current HttpSink partition transfer implementation.
…ooping over the messages of a partition in order not to loose messages.
…ly configured IDE anymore.
Codecov ReportAll modified lines are covered by tests ✅
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #3543 +/- ##
==========================================
+ Coverage 72.24% 72.45% +0.21%
==========================================
Files 863 867 +4
Lines 17306 17388 +82
Branches 986 988 +2
==========================================
+ Hits 12502 12598 +96
+ Misses 4392 4375 -17
- Partials 412 415 +3
☔ View full report in Codecov by Sentry. |
drcgjung
changed the title
drcgjung
changed the title
I'm waiting for @git-masoud to check before I undraft it. |
One point of discussion was the location of MultipleBinaryPartsDataSource.java I placed it under the testFixtures of data-plane-spi because it is not specific to just data-plane-http and could be used to test all kinds of pipeline sinks ... |
What this PR changes/adds
This PR introduces a test case to verify the loss of messages by HttpSink under various partitioning conditions.
This PR removes an early return statement in the partition transfer loop of HttpSink which leads to the loss of messages.
Why it does that
Currently, HttpSink looses messages even in completely deterministic settings. It may only be safely applied with partition.size=1
Further notes
Linked Issue(s)
Closes #3417
Please be sure to take a look at the contributing guidelines and our etiquette for pull requests.