Offset flag ignored

[ibalashov]

It looks like -o flag applies only up to kafka storage blocks, or some other caching is going on, e.g.

$ for i in {01..05}; do echo -n "$i " ; sh -c "~/kafkacat/kafkacat -C -b localhost:9092 -t topic -K\' -o 98564$i -p 0 -c 3 -u | md5sum" ; done
01 d1e6071308960d3d04c9ff911ef32516  -
02 d1e6071308960d3d04c9ff911ef32516  -

$ for i in {01..05}; do echo -n "$i " ; sh -c "~/kafkacat/kafkacat -C -b localhost:9092 -t topic -K\' -o 98574$i -p 0 -c 3 -u | md5sum" ; done
01 372ec3fd9d054638ff817f73a4c50007  -
02 372ec3fd9d054638ff817f73a4c50007  -

Messages were loaded in kafka in batches with snappy compression, latest kafkacat used.

[edenhill]

Huhm, weird. I can't reproduce it.

Can you remove the md5sum:ing and provide kafkacat with the new -O option to make it print offsets for each message. That should indicate if it is consuming the wrong messages.

[ibalashov]

Thanks for pointing out the big O flag. Here is what I got:

 $ kafkacat -C -b localhost:9092 -t topic  -o 9857401 -p 0 -c 2 -u -O
9857398 ...
9857399 ...

[edenhill]

Ouch, that doesn't look good.

I'll look into it, thanks!

[edenhill]

This is a librdkafka issue, will close this one, please monitor this instead:
confluentinc/librdkafka#170

[edenhill]

This is fixed in librdkafka master now, see confluentinc/librdkafka#170
Can you update and give it a go?
Thanks

[ibalashov]

The fix worked for me, thanks!