error="unknown attributes on uncompressed message 8"

[kevinconaway]

We are piloting this project out and are running in to this error when trying to consume from one of our brokers:

unknown attributes on uncompressed message 8

The error is being returned from fetches.Errors()

This particular broker cluster has the following settings:

version=2.2.1
inter.broker.protocol.version: "2.2"
log.message.format.version: "0.10.1"

We do not see the same issue on another broker cluster with the following settings:

version: 2.7.1
inter.broker.protocol.version: 2.7.1
log.message.format.version: "2.7"

I've tried passing different versions to the client via kgo.MaxVersions to no avail

Is there some other option that I should be setting here or is there any way to get more debug information about what is happening here?

[twmb]

I think this is a tiny bug in this client. The broker is returning LogAppendTime (bit 3) for a message, and the client is not expecting that. This is almost certainly an error introduced in a previous large refactoring, and was uncaught because the integration tests are on new Kafkas (message sets were removed in Kafka 0.11).

I can patch this quickly and tag a patch release.

[twmb]

If possible, can you try this commit? 5fee169

If not, I can tag that directly, but if you can check that it fixes things before the patch release, that'd be great!

[kevinconaway]

Thanks.

I was reading through the spec and noticed that LogAppendTime in bit 3 would cause the value to be 8. However both of the clusters have LogAppendTime set as message.timestamp.type

[twmb]

Is the new cluster using records, vs. the old message set format?

[kevinconaway]

If possible, can you try this commit? 5fee169

It looks like there is a compiler error here

Is the new cluster using records, vs. the old message set format?

I'm sorry, I don't understand the question

[twmb]

It looks like there is a compiler error here

That teaches me for thinking this was easy enough to not need to stash my other work, very sorry about that. I'll run the integration tests for the next commit just to be sure...

I'm sorry, I don't understand the question

The other cluster settings you posted show log.message.format.version: "2.7", which uses the "new" RecordBatch format, rather than the old MessageSet. The client isn't as sensitive on attributes for the newer format. Point is, if the new format has LogAppendTime, that'll be fine because the client doesn't choke as hard there.

[twmb]

6e639fb compiles & passes (as well as the followup unrelated commit), if you could check.

[twmb]

I'll check back in an hour, if I don't hear back I'll go ahead and tag :)

[kevinconaway]

Just deployed this out to both clusters. It is unfortunately still failing with the same error on the 2.2.1 cluster.

[kevinconaway]

This fails in this little example as well

My bit twiddling skills leave a lot to be desired but can't the 3rd bit for LogAppendTime be 0 or 1? If so, I think this code is only checking if its 0

[twmb]

Actually thinking about this further, per my understanding, bit 8 shouldnt be set at all. Bits 1 and 2 correspond to compression in message set v0 and v1, and then bit 3 corresponds to the timestamp type for v1. A value of 8 means bit 4 is set, which I know nothing about. I’ll take a look to see what I find related to this. It’s possible my understanding is slightly off, or worst case I can just relax this validation since it isn’t providing much.

…

On Tue, Sep 28, 2021 at 13:07 Kevin Conaway ***@***.***> wrote: Just deployed this out to both clusters. It is unfortunately still failing with the same error on the 2.2.1 cluster. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#90 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJZIJK6RGVIW3NZN77MWDTUEIG6TANCNFSM5E57LYBA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[kevinconaway]

A value of 8 means bit 4 is set, which I know nothing about.

Per the spec:

This byte holds metadata attributes about the message.

The lowest 3 bits contain the compression codec used for the message.

The fourth lowest bit represents the timestamp type. 0 stands for CreateTime and 1 stands for LogAppendTime. The producer should always set this bit to 0. (since 0.10.0)

All other bits should be set to 0.

If thats the case, it seems like it is only the fourth bit set so 0000 1000 -> 2^3 -> 8

[twmb]

Looking back at https://kafka.apache.org/documentation/#messageset,it appears I misunderstood their bits. I’m fairly sure I know the problem, and I can patch this in 20. I read bits 0 to 2 as the first two bits, not the first three, and that continued to make sense to me because Kafka never used more than the first two bits for compression. Thanks for bringing this up, I can patch this after lunch.

…

On Tue, Sep 28, 2021 at 13:16 Kevin Conaway ***@***.***> wrote: This <6e639fb#diff-c6763587c65f4655271050afcde2ba38165b48756f75072ab54f3093246efc61R1281> fails in this little example <https://play.golang.org/p/L7U_aHfKIJ1> as well My bit twiddling skills leave a lot to be desired but can't the 3rd bit for LogAppendTime be 0 *or* 1? If so, I think this code is only checking if its 0 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#90 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJZIJNLLY4EYT7FQW3YVPTUEIH75ANCNFSM5E57LYBA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[twmb]

I very much hope this commit works: 688d6fb
Given the prior success rate, I'm expecting the worse, but please let me know.

[kevinconaway]

🤞

Will take me ~30m to get it out

[kevinconaway]

Looks good!

[twmb]

Awesome, thank you! I've gone ahead and tagged this in v1.1.2. I'm going to close this for now but if anything else crops up please let me know!