Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Serverless Change Feed Processing with Azure Functions, is it possible to scale-out across multiple partitions? #34153
I noticed the docs mention the ability to scale out processing of change feed in parallel by partition. It is not clear to me how this works with serverless and Azure Functions. If I wire up the Change Feed trigger will I end up with parallel execution flow per partition? Ideally I would like to scale our solution by having an azure function triggering across all partitions in parallel but processing in sequence within the partition. So imagine 2 partitions A and B, combined change feed stream looks like this A1, B1, A2, B2, A3, B3. Partition A events could process at the same time as partition B events but I want to ensure that the order within the partition is maintained so A1, A2, A3 and B1, B2, B3. Is this possible when using the Change Feed binding on an Azure Function?
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.
Jun 28, 2019
rimman and unassigned
Jun 28, 2019
Let's say you deploy your Cosmos DB Trigger and it starts with 1 instance. A single instance will try to consume all Partition Key Ranges in parallel, one thread per Partition Key Range (see https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed-processor), so they are effectively being processed in parallel.
So let's say you have PK Ranges A, B, C, and D. You start with 1 instance. Instance 1 will process all 4 in parallel.
The amount of PK Ranges can grow dynamically as your storage and throughput needs grow, and the Trigger and dynamically adjust.