Sink partitioning

[yhrn]

Hi,

We are interested in having sink partitioning. Our use case is to maintain per key ordering at the sink after a partitioned reducer and still be able to have more than one sink pod. We want multiple sink pods so that we can roll out updates without completely stopping the pipeline output during the update, since this causes spikes in overall resource usage when catching up.

Manual partitioning is not practical as we can't easily control a rolling update across multiple vertices.

So we're hoping that this would be a feature that you would consider for Numaflow and that we can discuss the details implications here.

Original question on Slack @vigith

[vigith]

Manual partitioning is not practical as we can't easily control a rolling update across multiple vertices.

@yhrn if can do rolling update on one vertex at a time, this problem can be solved right?

[yhrn]

If you mean that we would be able to specify that updates need to roll out gradually across multiple verticies, in this case a number of sinks, then yes that would make it feasible to set up the partitioning manually I think.

I guess it would work like this.

1. Define n sinks identical except the name, e.g. sink-0, sink-1, .., sink-(n-1). Specify that updates must roll out gradually across all of these.
2. In the partitioned reducer calculate a sink partition based on the key and tag outgoing messages with this
3. Route messages from the reducer to one of the sinks based on the tag

It would be a little verbose to define the sinks but not a huge deal. But I'm curious, why would it be complex to implement first class support for this kind of partitioning directly in Numaflow?

[vigith]

But I'm curious, why would it be complex to implement first class support for this kind of partitioning directly in Numaflow?

It is something we plan to do, we have an issue for that #1677. Could you please take a look and let me know whether it makes sense to you?

[yhrn]

Yes, that makes perfect sense to me.

[vigith]

https://numaflow.numaproj.io/user-guide/reference/ordered-processing/ is available in the nightly build. Please try it out and let us know. You can find an example in https://github.com/numaproj/numaflow/blob/main/examples/41-ordered-processing.yaml.