Add consumer support for transparent gzip compression #11

Merged
merged 1 commit into from Aug 9, 2012

Conversation

Projects
None yet
2 participants
Contributor

ept commented Aug 9, 2012

The Scala Kafka producer implementation has support for gzip-compressing
individual messages or sets of consecutive messages, and the compression
is transparent to the Scala Kafka consumer implementation. It's very
convenient (just a matter of setting compression.codec=1 in the producer
config).

The compression codec of a message is indicated in the bottom two bits
of the 'attributes' byte of messages with magic == 1. This means it's
possible to add transparent compression support to consumers without any
configuration.

This commit adds compression support to the Ruby consumer. Because a
compressed message may actually contain more than one message inside it
(this makes compression more effective by grouping lots of small
messages into one big message), I had to move some of the parsing logic
from Kafka::Consumer to Kafka::Message.

Add consumer support for transparent gzip compression
The Scala Kafka producer implementation has support for gzip-compressing
individual messages or sets of consecutive messages, and the compression
is transparent to the Scala Kafka consumer implementation. It's very
convenient (just a matter of setting compression.codec=1 in the producer
config).

The compression codec of a message is indicated in the bottom two bits
of the 'attributes' byte of messages with magic == 1. This means it's
possible to add transparent compression support to consumers without any
configuration.

This commit adds compression support to the Ruby consumer. Because a
compressed message may actually contain more than one message inside it
(this makes compression more effective by grouping lots of small
messages into one big message), I had to move some of the parsing logic
from Kafka::Consumer to Kafka::Message.

acrosa added a commit that referenced this pull request Aug 9, 2012

Merge pull request #11 from ept/compression
Add consumer support for transparent gzip compression

@acrosa acrosa merged commit 9cc1220 into acrosa:master Aug 9, 2012

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment