Container App Jobs do not scale beyond 10-12 replicas concurrently reliably

[robrennie]

This issue is a: (mark with an x)

• [ x] bug report -> please search issues before submitting
• documentation issue or request
• regression (a behavior that used to work and stopped in a new release)

Issue description

When executing Container App Jobs using Event-driven scaling triggered by an azure-queue Job Scale rule, only 10-12 (or so) replicas will run concurrently at a time, even if the "max executions" is set to 100. If, for example, 20 messages are in the storage queue, 10-12 messages will be consumed and trigger replicas to be created, and the remaining will wait in the queue until one of the prior replicas completes and frees up a slot. Eventually, all 20 messages will be handled, but only the first 10-12 concurrently - all else are serialized.

Here's a screenshot of an execution log. All of these executions were triggered by the same batch of queue messages however notice the top two didn't start until another execution completed. Therefore, this is DOUBLING the entire time it takes to concurrently execute a batch of queue messages.

It's almost like there's some internal throttle that stops creating replicas regardless of what the "max executions" is set to (100 in this case).

Steps to reproduce

1. Create a Container App Job, using the Consumption plan, using Event-driven scaling triggered by an azure-queue Job Scale rule.
2. In my case, the Container App was a 1 CPU, 2 GB size.
3. Set "max executions" to 100, set polling interval to 5 seconds.
4. Add 20 messages at once to the storage queue.
5. Watch only 10-12 (I've seen 13 and even 17 recently) start immediately.
6. Check queue and see some still remaining.
7. Wait for all jobs to complete - 2x as long as it should take.

[vinisoto]

Rob Rennie (@robrennie) - did you check if there any messages in the System Logs?

[robrennie]

Vini Soto (@vinisoto) - yes. They look fine. The executions don't start - the messages stay in the queue - but once they do start the system logs are fine. The scaler is not scaling correctly or is stopping at around 10 or so. We are still only reliably starting 8 executions simultaneously.

In theory, we should be able to start 100 simultaneous jobs if Max. Executions is set to 100 and we queue up 100 messages, no?

[anthonychu]

Rob Rennie (@robrennie) Can you please share more details on the scale rule that you've defined? You can do this by running az containerapp job show -n $APPNAME -g GROUPNAME -o json and extract the scale rule, redacting any sensitive information.

Also please share some details on how the job container works. Assuming each execution runs a single container that processes a single message, it should do the following:

1. Retrieve a single message with a visibility timeout value that is longer than the time expected to process the message. Based on your execution history, you might want this to be longer than 40 minutes.
2. Process the message.
3. Delete the message from the queue.
4. Exit the process normally (exit code 0).

Sometimes, the behavior you're describing is caused by the job execution deleting the message before processing has completed. The scaler needs to know the total number of unprocessed messages in the queue to calculate how many new job executions to start. Deleting messages too early will impact the accuracy of this number.

[robrennie]

Anthony Chu (@anthonychu) - Thank you! I was not handling the message per your process. I was deleting the message as soon as it was pulled from the queue. My thinking was in case the container crashed, I did not want the message to become visible later and the job restarted. I didn't realize the queue message had to be "held open" (so to speak) while processing.

Thanks again - started several multi-hour jobs with 20+ replicas, worked like a charm. Hopefully this acts as good documentation for someone who does what I did!