Scaling lag in Functions on ACA

[raorugan]

Hi I am comparing scaling behavior for various .NET services flavors here https://github.com/KaiWalter/message-distribution#tldr-conclusionand see that there seems to be a gap, before Functions on ACA pick up items from Service Bus queues. Is that to be expected?

GitHub - KaiWalter/message-distribution](https://github.com/KaiWalter/message-distribution)

[raorugan]

@KaiWalter - lets track this discussion in this thread

[KaiWalter]

Is there a way to also scale Fun-on-ACA to minimal 1 scale unit?

[raorugan]

@kaibocai - Does this mean you want to set max scale instances to '1'? cc// @krishnajaju

[KaiWalter]

The pattern with the gap suggests that maybe the 2 other Functions need to be scaled up from 0 to something to pick up ingress from queues. Hence I would want to try with minScale=1

[krishna-kariya]

Hi @KaiWalter,
Thanks for raising the issue. Can you add some information about the experiment done to get these graphs? What do these graphs represent? Also, please share the expected behavior and the actual behavior.
Can you share the function app name?

[KaiWalter]

Sure @krishna-kariya

So the experiment is described and implemented in this repo - please check out README

Basically I am comparing scaling behavior between Function deployed as regular ACA container (src/func* C# projects and infra/func* templates) vs Function on ACA deployment (src/acaf* C# projects and infra/acaf* templates).

If you look at the graph above, I would expect the same scaling behavior: top graph shows func* variant scale increasing and descreasing as expexted and botton graph shows acaf* first scaling up the distributor/dispatcher part, then a gap and the scaling up receiver parts.

Container App names:

NAME	TYPE	LOCATION
kw-messdis2acafdistributor	Function App	West Europe
kw-messdis2acafrecvexp	Function App	West Europe
kw-messdis2acafrecvstd	Function App	West Europe
kw-messdis2funcdistributor	Container App	West Europe
kw-messdis2funcrecvexp	Container App	West Europe
kw-messdis2funcrecvstd	Container App	West Europe
kw-messdis2testdata	Container App	West Europe

Subscription Id I could share in a private conversation e.g. on Discord or email.

Is this information sufficient?

[krishna-kariya]

Hi @KaiWalter ,
I was looking into this issue. I found one difference between both deployments which is Container Apps with Functions image (func) has scale rule for servicebus scaler with messageCount as 100 while Azure Functions on Container Apps (acaf) uses Keda default settings currently. Default value for messageCount is 5. This can be a reason for difference you are observing.

One difference in code for both deployments is acaf uses dotnet-isolated runtime while func uses dotnet runtime. Can you run both deployments with same configuration, same code?

[KaiWalter]

@krishna-kariya I will convert func also to dotnet-isolated and be back here.

[KaiWalter]

@krishna-kariya with branch https://github.com/KaiWalter/message-distribution/tree/func-isolated I also changed the Functions in ACA Containers func variant to dotnet-isolated, but that had no effect.

func stilll peeking once

while acaf Functions on Container Apps shows this scaling gap

[krishna-kariya]

@KaiWalter, it looks like the default resources used by single Functions on ACA instance is more compared to resources used by single Container App instance with functions image.

Resource allocation for single instance:

• Azure Functions on ACA (Default)
  • cpu - 1
  • memory - 2.0Gi
• Azure Container Apps (Configurable)
  • cpu -0.25
  • memory - 0.5 Gi

Initially, some default numbers of nodes are allocated for any ACA environment. During scaling, ACA uses these nodes to create app instances. For container apps scaling, the default number of nodes are sufficient as it uses less cpu, memory per instance. For function apps scaling, the default number of nodes is not sufficient and thus, ACA environment requests more nodes in backend. After new nodes are available to ACA environment, it uses them to create remaining instances for Function app. It takes some time to fetch new nodes and create remaining instances, therefore, we see a gap in processing between both deployments.

[krishna-kariya]

@KaiWalter , Do you have any further questions?

[KaiWalter]

@krishna-kariya sorry for getting back so late - I was enjoying my summer vacation.
OK, the information you provided makes sense and explains the delay before scaling up.
@syneex this is something we need to consider - gets closer to the Functions on Service Fabric Containers behavior again.

[raorugan]

@KaiWalter for functions on ACA you will soon be able to configure CPU/memory using workload profile types (both Consumption or Dedicated) . With this you should be able to choose configs as you chose for ACA and get similar performance