Skip to content

bug: produce request encoding breaks with multiple partitions per topic #124

@klaudworks

Description

@klaudworks

Description:

When a producer sends records to more than one partition of the same topic in a single request, the message format doesn't match what Kafka clients expect. This causes parsing failures on both sides — real Kafka clients can't read what KafScale sends, and KafScale can't read what real clients send.

Example of when this happens:

A producer writes to partitions 0 and 1 of topic "orders" in the same produce request. This is completely normal — most Kafka client libraries batch records by topic and send multiple partitions at once. When this request goes through the KafScale proxy, the proxy re-encodes it in a slightly wrong format, and the broker can't parse it. Going the other direction, if a Kafka client sends this request directly to KafScale, the broker fails to parse it too.

With only one partition per topic (e.g. writing to just partition 0 of "orders"), everything works fine, which is why this wasn't caught by existing tests.

What's wrong:

In the Kafka protocol, each partition entry is followed by a metadata footer (called "tagged fields"). The correct layout looks like:

topic "orders"
  partition 0 → records → footer
  partition 1 → records → footer
topic footer

KafScale instead puts the partition footer once after all partitions, not after each one:

topic "orders"
  partition 0 → records
  partition 1 → records
  footer        ← only one, should be one per partition
topic footer

With a single partition these are identical (one partition, one footer either way), so the bug is invisible. With two or more partitions, the bytes are in the wrong order and parsing fails.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions