v2.2

This release contains many performance enhancements and internal code optimizations to the consumer side of zio-kafka. The Runloop, the heart of the Consumer, has been optimized to reduce the polling and streaming latency and be able to provide higher throughput, as well as less GC pressure. This results in a throughput that is 2 to 5 times higher.

The Runloop polls the underlying Kafka client for new data. It also applies backpressure for streams that do not require data by pausing partitions. When the stream needs more data, the partition is resumed. Previously the Runloop would poll every pollInterval. Now it polls continuously. When all streams stop processing, data polling stops so that the broker can detect that this Kafka client is stalled.

Several other changes to be aware of:

• OffsetBatch.merge was deprecated in favour of add.
• Consumer setting pollInterval was removed because polling is now continuous. We recommend you use ConsumerSettings(bootstrapServers) and the with* methods to build consumer settings to prevent future source incompatibilities.
• Consumer setting runloopTimeout was added. When the runloop has nothing to do for this period, it will fail.

Changes

• Increase default runloop timeout @erikvanoosten (#775)
• Give ConsumerSettings.runloopTimeout a default @erikvanoosten (#774)
• Fix restartStreamsOnRebalancing @erikvanoosten (#772)
• Improve readability @erikvanoosten (#773)
• Update scalafmt-core to 3.7.3 @github-actions (#769)
• Add better-monadic-for scalac plugin @guizmaii (#765)
• Improve readability @guizmaii (#763)
• Remove unused offsetRetrieval field from ChangeSubscription command @guizmaii (#764)
• Fix getConsumerGroupMetadataIfAny: It shouldn't access the low-level consumer directly @guizmaii (#762)
• Optimize offerRecordsToStreams code @guizmaii (#761)
• Optimise handlePoll code: Ensure that both runningStreams and newAssignedStreams are Chunks to take advantage of the optimised version of ++ when computing the updatedStreams @guizmaii (#760)
• Divide allocations: Instantiate the OffsetImpl as late as possible to avoid allocatiing one OffsetImpl and one TopicPartition for each consumed Record @guizmaii (#759)
• add test case for the runloop polling behavior @tnielens (#684)
• Optimise offerRecordsToStreams code @guizmaii (#756)
• Tune benchmark parameters @svroonland (#729)
• Review of PR #727: Remove unused Request::dataQueue @guizmaii (#747)
• Improve performances: Remove the aggregateAsync call @guizmaii (#751)
• Improve Logback configuration in benchmarks @guizmaii (#750)
• Review of PR #727 @guizmaii (#745)
• Fix ConsumersComparisonBenchmark: Avoid booting the Consumer instance in the bench, which could add noise in the bench results @guizmaii (#748)
• Remove buffered records @erikvanoosten (#744)
• ZStream based implementation of poll loop @svroonland (#736)
• Feed streams through queue instead of promise @erikvanoosten (#727)
• First seek then use for manual assignments @erikvanoosten (#726)
• Stabilize benchmark @svroonland (#714)
• Fix CI @guizmaii (#723)
• Optimise Runloop::run code @guizmaii (#721)
• Add profiling in the CI @guizmaii (#696)
• Init comparison benchmarks: compare kafka-clients vs zio-kafka consumers @guizmaii (#695)
• Rollbacks @guizmaii (#722)
• Update logback-classic to 1.4.6 @github-actions (#717)
• Update zio-sbt-ci, zio-sbt-ecosystem, ... to 0.3.10+94-cbe49d51-SNAPSHOT @github-actions (#718)
• Make as much computation as possible in fulfillRequests code @guizmaii (#720)
• Rename OffsetBatch::merge(Offset) to OffsetBatch::add(Offset) to improve readability @guizmaii (#700)
• Apply ZIO SBT Ecosystem and CI Plugins @khajavi (#702)
• Improve polling behavior @svroonland (#683)
• Update logback-classic to 1.3.6 @github-actions (#709)
• Update README.md @github-actions (#693)
• Consumer benchmark @svroonland (#689)

🌱 Dependency Updates

• Bump hmarr/auto-approve-action from 3.2.0 to 3.2.1 @dependabot (#734)
• Bump hmarr/auto-approve-action from 3.1.0 to 3.2.0 @dependabot (#705)
• Bump benchmark-action/github-action-benchmark from 1.16.0 to 1.16.2 @dependabot (#704)