Please sign in to comment.
Add consumer support for transparent gzip compression
The Scala Kafka producer implementation has support for gzip-compressing individual messages or sets of consecutive messages, and the compression is transparent to the Scala Kafka consumer implementation. It's very convenient (just a matter of setting compression.codec=1 in the producer config). The compression codec of a message is indicated in the bottom two bits of the 'attributes' byte of messages with magic == 1. This means it's possible to add transparent compression support to consumers without any configuration. This commit adds compression support to the Ruby consumer. Because a compressed message may actually contain more than one message inside it (this makes compression more effective by grouping lots of small messages into one big message), I had to move some of the parsing logic from Kafka::Consumer to Kafka::Message.
- Loading branch information...
Showing with 129 additions and 72 deletions.