Fix dataloss issue in restarting-streams-at-rebalancing mode

[vigoo]

Fixes #469

[svroonland]

Looks good! I wonder if we could simplify a bit by not buffering at all but immediately pushing the 'to be buffered' records on the partition stream with this mechanism.

[svroonland]

Should it be private?

[vigoo]

Looks good! I wonder if we could simplify a bit by not buffering at all but immediately pushing the 'to be buffered' records on the partition stream with this mechanism.

Sounds like a good idea but:

• it will also affect the "normal mode" - need to think about whether it remains correct or not.
• I'm not sure it would not make the processing unordered

[vigoo]

Note: I'm still trying to validate this in production and seeing some problems (not sure yet if related)

[vigoo]

The problem was that merge introduced a 1-element buffer for each partition (at least the ZIO 1 version) and it was taking too much memory in our application. As the whole drainQueue was only used at the "end" of the partition stream anyway, I changed the merge to concat which fixed our memory issue.

With this change from my end the change is ready to merge

[vigoo]

Note: one more fix is required for this, I'm validating it in prod now.

[vigoo]

I think the fix is now complete, I plan to publish a detailed blog post about it that will help understand.

[vigoo]

Detailed explanation: https://ziverge.com/blog/zio-kafka-with-transactions-debugging-story