Stop scaling out new resources if the function isn't successfully processing

[jeffhollan]

There are two scenarios we have today where we stop scaling new instances even as the queue length increases:

1. No available partitions. If you have an event hub with 5 partitions that is growing in event lenghts, we will scale to 5 instances, and then 6. The 6th one gets scaled out but can't get a lock on a partition so never actually consumes any events. We never scale to a 7th because we look to see that the 6th isn't execution so stop scaling.
2. A function is misconfigured and never actually is able to start consuming. Rather than scaling out a broken app, we stop scaling if we see no executions are happening.

In our service we do that by looking at the execution and billing metrics for the instance, and if no billing or execution metrics are emitting we stop scaling more.

We'd need a similar pattern in Kore so that if the event consumer isn't actually processing new messages, we don't keep scaling. Both for the "no available partition" scenario or the "my app isn't even working" scenario. We could also solve each of these in 2 seperate ways (e.g. maybe expose partition info into the scaler?)

[markusthoemmes]

Would that be solvable via more advanced metrics as have been mentioned on the last call? We could use a consumptionRate metrics to detect this without having having to leave the metrics context of the source for autoscaling (queueLength and consumptionRate would be two metrics generated by the same source).

That would also nicely dovetail into using these metrics for scale > 1.

[gabrielSoudry]

Some news ? The first case "No available partition" is common.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.