azure pipeline agents don't always remove

[BenH-Puregym]

Report

i've got scaledjob setup and a placeholder agent in ADO. Sometimes the agents are remove when their job is finished and sometimes they're not. Some agents which don't get removed will have a job lasting 13 minutes and others 1hr so doesn't seem to be consistent.

Expected Behavior

the agents should remove each time a job has finished

Actual Behavior

some do, some don't. leaving the pool with tons of offline agents.

Steps to Reproduce the Problem

1. setup a pipeline scaledjob with 10/20 containers
2. run a unit test pipeline or similar which has 10 parallel jobs
3. repeat a few times and you will see the agents being left.

Logs from KEDA operator

example

KEDA Version

No response

Kubernetes Version

No response

Platform

Microsoft Azure

Scaler Details

pipeline

Anything else?

No response

[JorTurFer]

hey,
Is the agent pod finishing? I mean, KEDA doesn't know if the pod has finished or not its work, KEDA removes the pod/job once it has finished with success or failure, so if your pod is still alive without finishing (like a job should do), KEDA will not remove it

[BenH-Puregym]

Hey @JorTurFer , So the agent is left in an offline state in the ADO agent pool but when you check back in AKS, the pod and job is no longer there or the job shows as success. Then requires to be manually removed from ADO.

[JorTurFer]

Just to clarify it, is it your problem related with agent pods not deleted from the AKS or with orphan agent in Azure DevOps?

[BenH-Puregym]

maybe would be best for me to explain the problem again:

1. job is run on ADO
2. keda starts jobs and adds pods to the ADO pool
3. job finishes on ADO
4. keda deletes the pod and stops the job
5. agents still exist in ADO pool as offline

[tomkerkhove]

So what @JorTurFer says in #3349 (comment) is correct - KEDA just adds instances to an existing ADO pool to spin up pipelines based on the current pipeline queue.

However, to be sure - Would you mind adding some screenshots from Azure DevOps/Pipelines to see if it works as intended please? Ideally just of your ADO pool and its overview

[BenH-Puregym]

so perhaps a problem with the script that MS have provided here which is supposed to remove the agent when the job is finished rather than a problem with Keda. Like I say, it work sometimes but doesn't always.

screenshot as requests:

as you can see, some of them didn't even run a job before going offline.

[tomkerkhove]

That seems like a agent problem rather than KEDA indeed - Thanks for sharing!

Once a job completes, it should remove it from the pool indeed.

[JorTurFer]

hey,
I got it. KEDA doesn't handle ADO at all, the only goal of KEDA is to make the autoscaling dead simple, but not to manage the upstreams.
I mean, in the same way that you register the agent by yourself in AzDO (it does automatically on pod agent) you should unregister it. My suggestion is to unregister the agent after executing the pipeline, before finishing the job execution.

[kamilpi]

Hi,

I think you do not need to unregister it from ADO.

I leave it here for future similar issues.
It depends on how you build the image of the ADO agent.
According to the documentation you can find the flag that will shut down the agent after the build is executed: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops#start-the-image

Script start.sh:

./run.sh --once "$@" & wait $!

Enjoy!