You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've been taking a look at the source in consumer.d within processBuffer. It seems kafka-d only handles the older message format (prior to Kafka 0.11) which are structured like:
MessageSet (Version: 0) => [offset message_size message]
offset => INT64
message_size => INT32
message => crc magic_byte attributes key value
crc => INT32
magic_byte => INT8
attributes => INT8
bit 0~2:
0: no compression
1: gzip
2: snappy
bit 3~7: unused
key => BYTES
value => BYTES
newer versions of Kafka use the following structure which includes a timestamp as well as also supporting lz4 compression.
MessageSet (Version: 1) => [offset message_size message]
offset => INT64
message_size => INT32
message => crc magic_byte attributes timestamp key value
crc => INT32
magic_byte => INT8
attributes => INT8
bit 0~2:
0: no compression
1: gzip
2: snappy
3: lz4
bit 3: timestampType
0: create time
1: log append time
bit 4~7: unused
timestamp => INT64
key => BYTES
value => BYTES
The text was updated successfully, but these errors were encountered:
I've been taking a look at the source in consumer.d within processBuffer. It seems kafka-d only handles the older message format (prior to Kafka 0.11) which are structured like:
newer versions of Kafka use the following structure which includes a timestamp as well as also supporting lz4 compression.
The text was updated successfully, but these errors were encountered: