Repository navigation
Client-defined key affinity on scalable topics (equivalent of v4 Key_Shared STICKY hash ranges)? #27617
Replies: 1 comment
|
Hi, thanks for the detailed comparison — you've read the spec correctly, and I verified the two points that decide your questions. The short answer is: no, v5 has no way for a consumer to declare the hash ranges it owns — and yes, for your use case, staying on v4 Key_Shared STICKY over persistent topics is the intended pattern for now. What the spec actually says (
So for question 2: yes — if fixed keyspace ownership that survives graceful restarts is a hard requirement (stateful actor semantics), stay on v4 Key_Shared STICKY over persistent topics, where the client declares |
Uh oh!
There was an error while loading. Please reload this page.
In v4, a Key_Shared consumer in STICKY mode could declare its own hash ranges. Whatever happened on the broker side (partition count changes, other consumers joining or leaving), that consumer was guaranteed to receive every message whose key hashed into its ranges. We rely on that: each consumer instance owns a fixed slice of the keyspace.
In v5, the spec gives Stream consumers per-key affinity through entry-bucketing (PIP-486, spec/scalable-topics/key-shared.md). But bucket ownership is assigned by the controller, and java-mapping.md says entry-bucketing has no application-facing API. So a consumer can’t say “I own hash range X–Y.” We've tested the stickiness with v5 and it really only shows up in a grace period when failure happens (not when a client gracefully disconnects) and only when the number of segments matches or exceeds the number of consumers. Shared segments don't appear to have any stickiness at all.
I'm asking because our system has stateful actor-like semantics and in certain cases, it is expensive for keys to be reassigned elsewhere and then reassigned back (e.g. during a release as processes come and go gracefully).
Questions:
All reactions