Compression / Buffer issue

[MrLotU]

Hi, I've been trying to use SwiftKafka for a project of mine, but run into an issue when sending messages of around 1.5KB. Instead of the JSON I try to send, it sends some malformed data like this:

2t_id":"16315612"??????

followed by the rest of the JSON in the correct format. Because of this the JSON becomes invalid and unusable.

I've tried multiple config options to increase buffer sizes to allow for bigger messages, but so far nothing has seemed to work. The options I've tried is setting

config.messageMaxBytes = 1000000000
config.socketAendBufferBytes = 1000000000

If I do this however, the message is still malformed and I get the following error in my console:

Failed to set socket send buffer size to 10000000: No buffer space available

Any and all help is much appreciated!

[MrLotU]

After @ianpartridge's comment on Slack regarding C library interfacing, I did some poking around myself and found that changing this line from RD_KAFKA_MSG_F_COPY to RD_KAFKA_MSG_F_BLOCK, the issue goes away.

I have no idea what those flags do and mean, other than the explanation from the librdkafka header here. I am therefore not sure if this is a change that'd be open to be PRed. Would love to have some feedback. Thanks!

[winsmith]

I made my own fork with this change and a note regarding installation at https://github.com/TelemetryDeck/SwiftKafka

Sadly, the change has not helped. In fact, every 10th message or so is now completely garbled. I think at this point I'll abandon SwiftKafka (and Kafka as a message queue) in favor of something more maintained.

(I'm not expecting anyone to read this, but if you stumble in here in the year 2048, looking for a solution, say hi! Sorry for not solving climate change.)