-
Notifications
You must be signed in to change notification settings - Fork 12
Description
Description:
When a producer sends records to more than one partition of the same topic in a single request, the message format doesn't match what Kafka clients expect. This causes parsing failures on both sides — real Kafka clients can't read what KafScale sends, and KafScale can't read what real clients send.
Example of when this happens:
A producer writes to partitions 0 and 1 of topic "orders" in the same produce request. This is completely normal — most Kafka client libraries batch records by topic and send multiple partitions at once. When this request goes through the KafScale proxy, the proxy re-encodes it in a slightly wrong format, and the broker can't parse it. Going the other direction, if a Kafka client sends this request directly to KafScale, the broker fails to parse it too.
With only one partition per topic (e.g. writing to just partition 0 of "orders"), everything works fine, which is why this wasn't caught by existing tests.
What's wrong:
In the Kafka protocol, each partition entry is followed by a metadata footer (called "tagged fields"). The correct layout looks like:
topic "orders"
partition 0 → records → footer
partition 1 → records → footer
topic footer
KafScale instead puts the partition footer once after all partitions, not after each one:
topic "orders"
partition 0 → records
partition 1 → records
footer ← only one, should be one per partition
topic footer
With a single partition these are identical (one partition, one footer either way), so the bug is invisible. With two or more partitions, the bytes are in the wrong order and parsing fails.