Let's Take a Look at... KIP-932: Queues for Kafka! - Gunnar Morling

[giscus[bot]]

Let's Take a Look at... KIP-932: Queues for Kafka! - Gunnar Morling

In the "Let’s Take a Look at…!" blog series I am going to explore interesting projects, developments and technologies in the data and streaming space. This can be KIPs and FLIPs, open-source projects, services, and more. The idea is to get some hands-on experience, learn about potential use cases and applications, and understand the trade-offs involved. If you think there’s a specific subject I should take a look at, let me know in the comments below!

That guy above? Yep, that’s me, whenever someone says "Kafka queue". Because, that’s not what Apache Kafka is. At its core, Kafka is a distributed durable event log. Producers write events to a topic, organized in partitions which are distributed amongst the brokers of a Kafka cluster. Consumers, organized in groups, divide the partitions they process amongst themselves, so that each partition of a topic is read by exactly one consumer in the group.

https://www.morling.dev/blog/kip-932-queues-for-kafka/

[masoudniki]

In your writing, you mentioned that "it’s currently not supported to use share groups and thus Kafka queues when reading from a follower node in the Kafka cluster." Could you clarify whether this means that if we have replication enabled in our Kafka cluster, we cannot use Kafka queues at all? Or is this limitation specifically related to consumers reading from follower nodes instead of leaders?

[gunnarmorling]

Or is this limitation specifically related to consumers reading from follower nodes instead of leaders?

Yes, it's exactly that, you currently can't use share groups when consuming from a replica (KIP-392).

[JRobTS]

Excellent post.

I feel like to solve head-of-line blocking you really need the DLQ. Without it, consumers may as well simply accept every message whether they use it or not.

KIP 932 seems to solve some issues around matching partition count to consumer count, which is necessary but insufficient for adoption. Adding partitions to a topic really isn't that hard.

I completely agree that DLQ support and retry timing are necessary additions. Without them I don't see much adoption of queues.

[gunnarmorling]

Yeah, I definitely hope these will be added rather sooner than later.