fetchMaxBytes

[mickhansen]

I'm using HighLevelConsumer with autoCommit: false, for each message i'm pushing it into an internal queue so i can parse them serially and then comitting the offset after all messages are parsed.

However i'm having an issue with fetchMaxBytes. If set to fetchMaxBytes: 1024 * 128 it works but i get a queue of 150 messages.

If i set fetchMaxBytes to something lower, 1024 * 64 or 1024 * 90 i don't get any messages at all.
Any help would be appreciated, ideally i'd like less messages at a time so i can get more often commits in case of errors.

[haio]

@mickhansen is this issue related to #181 ?

[mickhansen]

@haio Not directly. I implemented what you suggested. I'm however seeing issues with fetchMaxBytes as described.

[haio]

it should because the size single message is bigger than 1024 * 90, so the consumer can't ever get a complete message.

[mickhansen]

@haio That doesn't really correspond with the fact that it will fetch 150 mesages at 1024 * 128 with autoCommit false, if 150 messages fit in 1024 * 128 i find it hard to believe that a single message couldnt fit in 1024 * 90, especially considering all my messages are about 3-4kb compressed json.

[haio]

@mickhansen that's interesting, I need to reproduce it.

[haio]

Just made a test, in my test fetchMaxBytes is 1024*5 and sent exactly 4kb message to kafka, consumer can get all the message, if fetchMaxBytes is less than 1024*5, it can't consume any message.

[mickhansen]

And getting one message at the time?
Any way i can measure the size of my offsets in kafka?

[haio]

yeah, it should be one message one request. you can use offset.fetch({topic: , partition: }) to get available offset of the partition.

[ginokurian]

Somewhat off this topic, when we mention that fetchMaxBytes: 1024 * 128 does it mean that we are telling the consumer to fetch 128 messages of length 1kb each??..or does it just mean that it will fetch 128kB of messages....
I found the documentation lacking a bit of clarity in this...

[mickhansen]

@ginokurian No, that tells the consumer to fetch 128KB in total, however many messages that might be