Is Consumer Group prunes old members?

[abishai]

I have an urge to use NATS instead of Kafka for our next project, however I need to replicate kafka partitions (sequential processing of related messages and parallel procession of unrelated).

I read readme.md and unit tests, however I'm unsure about how it will scale up and down if no stable ordinal available.

For example, pod 1 starts and configures elastic consumer group with size = 10, running 1 consumer.
If I start another instance, it joins with different consumer name.
Can I have random consumer names in this scheme, because I don't have ordinal, like pod-1, pod-2, I have pod-, so every time deployment scales, it will join with new name. Is NATS cleans stale consumers or orphaned work queues will hang indefinitely?

[jnmoyne]

I'm assuming you are talking about the partitioned consumer group feature in Orbit.

It doesn't do automatic member management: even for elastic consumer groups the membership must be managed administratively (or programmatically). The reason being that it is difficult for it to do that management automatically and if you want automated management of elastic consumer group's membership it is better to do that at the process (or container) management level (which has an authoritative view of which processes or pods are running or not. So it's elastic in that you can add remove members at any time but it doesn't do that automatically. If you can have something get invoked each time you start or stop a pod (or on a periodic schedule) then that can use the library (or invoke the cg CLI tool) to automatically add/remove members.

We may still add a new feature in the future to consumers in the server that would allow what you want (think being able to set a max acks pending value per subject or per hierarchy of subjects) and yet still be a JetSteam Consumer, meaning no need for partitions and client applications can start or stop pulling from the consumer at any time without having to manage a membership, but at the moment that is not there yet.

[abishai]

I see, so I need a stable numbers for pods and add them in partitioned consumer group? For example, if I have 3 pods in deployment, I need that pod impersonates itself as 1, another as 2 and the another as 3. When the pod starts, it derives valid member name (like consumer-1, consumer-2, consumer-3) and start consuming?

[jnmoyne]

With elastic groups the number doesn't have to be stable, but you need to administratively (or programmatically) manage the group. If you want to set the group's membership once and never change it again then yes it would need to be a stable number of members. And yes the pods should have consistent member names (i.e. like you describe).

You could do something where you have the list of member names and pods as they come out pick one of the member names in the list and use it to join the group. You can have more than one pod with the same member name, and in that case all the pods with the same member name become a fault-tolerant group (one of them gets picked automatically as the 'pinned' instance and get the messages, while the other are in stand-by, as soon (the timing is controlled by the 'ack wait' value) as the currently pinned instance stops (or gets killed, or crashes) another one becomes the pinned one and starts getting the messages).