sarama drops message when either gzip/snappy is used

[amalakar]

I wrote a benchmark program to see performance of the library. I noticed that it drops significant number of messages when either gzip/snappy is used with waitForLocal.
I get around 59.5 MBps without compression

I see the following error message in case of gzip/snappy:

[Sarama] 2014/03/11 18:39:06 kafka: Dropped 35497 messages
[Sarama] 2014/03/11 18:39:06 kafka: Dropped 35497 messages
[Sarama] 2014/03/11 18:39:06 kafka: Dropped 35497 messages
[Sarama] 2014/03/11 18:39:06 kafka: Dropped 35497 messages

I am sending 10 million messages, with gzip/snappy, 65k buffer and 1 second buffer timeout. Also eventually the benchmarking program gets killed.

[patricklucas]

We're seeing this as well. Compression seems to work fine for our low-volume logs but as soon as we hit a few thousand messages per second it starts to drop lines.

Worse, the messages appear to be reenqueued and begin to stack up, eventually OOMing.

[eapache]

It's not clear to me if this is simply a performance issue (where our compression path is slow and we fall behind) or an actual error. The fact that messages are reenqueued is what's throwing me - that only happens when the kafka broker returns us an error, and if it doesn't like the messages we're sending I would expect that to show up even at low volumes.

Can somebody experiencing this take a brief traffic capture when this occurs and see what the broker is actually sending us on the wire? 1.12 prereleases of Wireshark are capable of dissecting kafka traffic if you tell it which port your broker is on.

[eapache]

Additionally, the change I just merged should give us more detailed log messages which might help pinpointing the problem.

[eapache]

Just merged a new producer design which should fix this issue among many others.