Allow "topic claim" mode

[zorkian]

Marshal should support an option where you can specify that you want a claim to be on the entire topic. This is used to support the sharded production use case, where you want to ensure all messages in a given shard go to a given consumer.

[zorkian]

cc @DrTall @basharal

I started working on this tonight but it ends up being slightly less trivial than I expected.

If we just claim partition 0, that's fine, but we need to produce/store offsets to the other partitions too. Marshal won't let us track them without major refactoring since it requires (and has safeguards) about not consuming from/committing offsets to an unclaimed partition.

We can make it so that it claims the rest of the partitions after claiming 0, but this definitely ends up in a state where someone else might have the other partitions briefly and we fail to claim them because they haven't yet expired (but 0 expired already -- heartbeats are jittered so we could have 0 expire first). It's not too hard to make it retry the claims in that case, though.

In this mode should the health check still work or should we disable releasing unhealthy topics? If we can't handle all of the traffic from the partitions then what? Print some major errors and tell the user they've exceeded the ability to do sharded consumption?

There's a lot of places that have to be updated to make this work. I still think the claim-0 mode makes sense, though, rather than trying to expand the protocol to have a ClaimTopic message type. I will keep looking at it.

[DrTall]

Maybe I'm confused, but can't we just use Kafka's regular offset tracking
for the nonzero partition messages? I mean basically returning some other
message whose "commit" function doesn't go through the Marshal flow but
rather just dies the optio pay commit?

This comment might be more cogent if I were.
On Nov 24, 2015 9:29 PM, "Mark Smith" notifications@github.com wrote:

cc @DrTall https://github.com/DrTall @basharal
https://github.com/basharal

I started working on this tonight but it ends up being slightly less
trivial than I expected.

If we just claim partition 0, that's fine, but we need to produce/store
offsets to the other partitions too. Marshal won't let us track them
without major refactoring since it requires (and has safeguards) about not
consuming from/committing offsets to an unclaimed partition.

We can make it so that it claims the rest of the partitions after claiming
0, but this definitely ends up in a state where someone else might have the
other partitions briefly and we fail to claim them because they haven't yet
expired (but 0 expired already -- heartbeats are jittered so we could have
0 expire first). It's not too hard to make it retry the claims in that
case, though.

In this mode should the health check still work or should we disable
releasing unhealthy topics? If we can't handle all of the traffic from the
partitions then what? Print some major errors and tell the user they've
exceeded the ability to do sharded consumption?

There's a lot of places that have to be updated to make this work. I still
think the claim-0 mode makes sense, though, rather than trying to expand
the protocol to have a ClaimTopic message type. I will keep looking at it.

—
Reply to this email directly or view it on GitHub
#2 (comment).