Big messages disappearing

[frranck]

Hello,
If I'm writing a "too big" message with

err = w.WriteMessages(context.Background(),
kafka.Message{
Key: key,
Value: value,
},
)
if err != nil {
log.Errorln("WriteMessage", err.Error())
}

I'm not getting any errors, kafka and zookeeper don't show anything in the logs, and the message simply doesn't appear with kafka-console-consumer.sh.

Any idea how to fix/troubleshot that ?

[achille-roussel]

Is the writer configured to produce asynchronously to kafka?

If that’s the case it’s expexted to be a “lossy” production mode, messages that failed to be delivered will be retired, or dropped if they failed because of a non-recoverable error (like in your case).

[frranck]

@achille-roussel thanks a lot, that helped me to get an error.

WriteMessage [10] Message Size Too Large: the server has a configurable maximum message size to avoid unbounded memory allocation and the client attempted to produce a message larger than this maximum