ScaledObject accurate scaling strategy

[audunsol]

Hello,

Is there something analogous to the

  scalingStrategy:
    strategy: "accurate"

for ScaledObjects, like we have for ScaledJobs?

Some more context on my issue:

We have been using KEDA ScaledJobs for some heavy simulations for several years now. Setup:

• Trigger: Azure Storage Queue
• scalingStrategy: accurate
• queueLength: '1'
• Processing time for a message is usually around 5 and 120 seconds (previously often up to 10 - 20 minutes, which may occasionally still happen, depending on calculation input parameters)
• Messages are deleted when read, before the heavy calculation/processing starts.
• Handling SIGTERM in the app code after message is read by ignoring it until calculation is done, and have a termination grace period of 1 hour

This has worked OK, but over time the simulation performance has improved, and hence the cold start penalty of starting a new pod for every message has become a real pain point (end users are effectively the ones putting these messages on the queue from our UI, and watching progress bars fed by updates from the simulatsions via WebSockets etc.).

So I am trying to improve the situation by running our simulations as Deployments instead, using ScaledObjects. Running minimum 3 replicas always-on seems to fix the perceived cold start for a lot of the use cases. So effectively, instead of scaling to 0, we are now throwing resources/money at the problem, which is an OK tradeoff at this time).

The main issue now is that if we push 4 messages to the queue, 3 simulations starts "immediately", but the 4th one does not start until one of the previous ones are done. If pushing 10 messages to the queue, it scales out fine to e.g. 7 replicas (since the queueLength is very quickly reduced to 7 when the first 3 are deleted), but then the 3 last ones have to wait for the other replicas to be done processing, and ready to pick up a new message. All messages are processed "eventually" with this setup.

This is not an issue we had with the ScaledJob approach. We got new jobs started until the queue was empty, after we applied the accurate scaling strategy. And this is basically what we want with the ScaledJobs as well.

I am not able to find a way to configure this with KEDA ScaledJobs/HPA. Am I missing something out?

[JorTurFer]

Hi,
I understand that you want to scale faster, right? You could use the advanced section to define the HPA behaviour and scale faster https://keda.sh/docs/2.10/concepts/scaling-deployments/#advanced

One question, do you use deployment ony for having always some active pods in order to avoid cold starts? I ask because ScaledJob has minReplicaCount that you can use for having always some jobs ready to process incoming messages. If you are comfortable using ScaledJobs, maybe it's a better approach. I assume that your process isn't so fast, so using ScaledObject you can be affected by arbitrary pod removal during the scale in process, meanwhile ScaledJobs aren't affected by this

[audunsol]

Hello,

Thanks for your reply @JorTurFer!

The more I think of this issue, the more I realize that the main problem is our sequence of:

1. Read message
2. Delete message
3. Process message
4. Write result to blob

If we change it to:

1. Read message
2. Process message (while renewing lease by extending visibility timeout on the message until done)
3. Write result to blob
4. Delete message

...the queue length would reflect both unprocessed and ongoing messages, and the scaler would be able to scale out to 1 replica per message in parallel, which is what I want (and my intention on the original question).

I see now that #4002 adds a fix to make this behave exactly the way I want (as long as I update the application code to stop deleting the messages before they are processed).

Otherwise, there is no real way for KEDA/HPA to know the difference between a "busy" and an "idle" Pod when running in a Deployment like this, so I think there is no way for KEDA/HPA to know that the remaining messages in the queue are being processed by the running pods or not. Jobs, on the other hand, can be assumed to be busy until they complete and exit, so if we have more messages in the queue, KEDA should simply start more Jobs.

Regarding your comment @JorTurFer:

ScaledJob has minReplicaCount that you can use for having always some jobs ready to process incoming messages

Thanks for the suggestion, I didn't really think of that option. It can act a bit weird when deciding when to complete a job (e.g. "done polling for messages, nothing in queue" would no longer be a proper exit condition). Not necessarily a big deal, and possible to work around I guess. I might actually give this a shot, as it seems easier on short terms than rewriting the application code logic as described above 👍

I assume that your process isn't so fast, so using ScaledObject you can be affected by arbitrary pod removal during the scale in process

Yes, this is the reason we went for ScaledJobs in the first place.

However, our views on the simulation runs have changed a bit after setting up this, so some reasons we would prefer ScaledObjects instead now are (just listing in case it is of any interest for original question background/context and our KEDA end user perspective):

• Some efforts on the application/calculation code has made many of the simulations faster (and the very long running jobs are now more like outliers, even though they still happen depending on input data payload parameters). Waiting several seconds for a job pod to start processing a 5-10 second simulation is now less acceptable, compared to a 2-20 minute simulation.
• Our container images are somewhat on the heavy side (NodeJS based), and combined with cluster autoscaling we are also spending some time on image pulling (particularly on newly started k8s nodes) and startup/init time. Re-using the same container for multiple messages, and keeping it running seems better.
• We have more paying customers, so having constantly running pods (and more machines) is becoming a smaller concern compared to unhappy users.
• We see that our usage patterns are bursty, but when users are working in these areas of our application, it will generate calculation tasks more frequently, so the cooldown period of ScaledObjects/HPA fits very well with this pattern (user tweaks some parameters and re-runs calculations, and now the pods are still running warm and ready for processing a new burst, as they didn't get time to scale down first).
• We have spent some effort into handling and testing termination signals (ignoring them until calculation is done), and setting pod shutdown grace period to a long time, so these cases are more or less handled as far as I can see now.

[JorTurFer]

Thanks for update the thread with the changes :)
Happy to see that you have found a working solution ❤️