[BUG]: Agent fails during "Initialize Job" with "Access to path '/agent/_work/_tasks/_temp_xxx is denied.'

[dieter-adesso]

What happened?

Sporadically, our Pipelines are running on error during the "Initialize Job" due to "Access to path '/agent/_work/_tasks/_temp...' is denied." error. When it happens, every subsequent pipeline running on this agent will fail until the VM gets stopped (e.g. after 30 minutes idle).

Our environment is setup with multiple Pipelines (Build & Deploy) for different topics. Especially the Deploy Pipelines for one specific topic triggers up to 10 Deploy-pipelines in parallel.
The VMSS Pool is configured for max 4 servers.

Activated system.debug and agent.diagnostics, but there was no helpful information.

When the issue occured, we connected to the agent and see, that the path /agent/_work/_tasks/ was owned by root root drwxr-xr-x and not by AzDevOps AzDevOps drwxr-xr-x.
The checked folder itself was empty.
Verification on other agents show this folder is owned by AzDevOps.

You can find a raw log attached. It is from a small Test-Pipeline, running into the same issue.

Since owner of this path is root and our Pipelines are running as AzDevOps, I see this as clear Agent bug.

It is important to note, that this has severe impact on progress of our development team!

Versions

agent release 4.254.0

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Ubuntu 22_04-lts-gen2

Version controll system

No response

Relevant log output

>> Agent raw logs <<
2025-04-10T14:32:34.4276879Z ##[section]Starting: Version - DEV
2025-04-10T14:32:34.4390581Z ##[debug]Cleaning agent temp folder: /agent/_work/_temp
2025-04-10T14:32:34.4480645Z ##[debug]Skipping overwrite %TEMP% environment variable
2025-04-10T14:32:35.3761164Z ##[section]Starting: Initialize job
2025-04-10T14:32:35.3763840Z Agent name: 'vtrevamp-devops-buildagent0000TS'
2025-04-10T14:32:35.3764705Z Agent machine name: 'vtrevamp-devops-buildagent0000TS'
2025-04-10T14:32:35.3765088Z Current agent version: '4.254.0'
2025-04-10T14:32:35.5665747Z ##[debug]Agent environment resources - Unable to get disk info, exception: Value cannot be null. (Parameter 'driveName'), Memory: Used 915.00 MB out of 15979.00 MB, CPU: Usage 4.61%
2025-04-10T14:32:37.0207546Z Agent running as: 'AzDevOps'
2025-04-10T14:32:37.0236782Z ##[debug]Triggering repository: VT.Corporate.CICD-Pipelines. repository type: Git
2025-04-10T14:32:37.0239324Z Prepare build directory.
2025-04-10T14:32:37.0426320Z ##[debug]Delete existing artifacts directory: '/agent/_work/4/a'
2025-04-10T14:32:37.0427654Z ##[debug]Deleting artifacts directory: '/agent/_work/4/a'
2025-04-10T14:32:37.0434907Z ##[debug]Creating artifacts directory: '/agent/_work/4/a'
2025-04-10T14:32:37.0436733Z ##[debug]Delete existing test results directory: '/agent/_work/4/TestResults'
2025-04-10T14:32:37.0437115Z ##[debug]Deleting test results directory: '/agent/_work/4/TestResults'
2025-04-10T14:32:37.0438696Z ##[debug]Creating test results directory: '/agent/_work/4/TestResults'
2025-04-10T14:32:37.0463844Z Set build variables.
2025-04-10T14:32:37.0477474Z Download all required tasks.
2025-04-10T14:32:37.0531757Z Downloading task: Bash (3.250.1)
2025-04-10T14:32:37.0632906Z ##[error]Access to the path '/agent/_work/_tasks/_temp_81eacb72-14d1-4f3a-9d2a-46f4d3c0bc5b' is denied.
2025-04-10T14:32:37.0639280Z ##[debug]System.UnauthorizedAccessException: Access to the path '/agent/_work/_tasks/_temp_81eacb72-14d1-4f3a-9d2a-46f4d3c0bc5b' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.FileSystem.CreateDirectory(String fullPath, UnixFileMode unixCreateMode)
   at System.IO.Directory.CreateDirectory(String path)
   at Microsoft.VisualStudio.Services.Agent.Worker.TaskManager.DownloadAsync(IExecutionContext executionContext, TaskStepDefinitionReference task) in /mnt/vss/_work/1/s/src/Agent.Worker/TaskManager.cs:line 212
   at Microsoft.VisualStudio.Services.Agent.Worker.TaskManager.DownloadAsync(IExecutionContext executionContext, IEnumerable`1 steps) in /mnt/vss/_work/1/s/src/Agent.Worker/TaskManager.cs:line 89
   at Microsoft.VisualStudio.Services.Agent.Worker.JobExtension.InitializeJob(IExecutionContext jobContext, AgentJobRequestMessage message) in /mnt/vss/_work/1/s/src/Agent.Worker/JobExtension.cs:line 144
2025-04-10T14:32:37.0641408Z ##[section]Finishing: Initialize job
2025-04-10T14:32:37.0670825Z ##[section]Finishing: Version - DEV

>>end Agent raw logs <<

>> ssh connect to agent <<
vtrevamp-admin@vtrevamp-devops-buildagent0000TS:/agent/_work/_tasks$ ps -ef| grep Az
root        4650       1  0 13:24 ?        00:00:00 sudo -E nice -n 0 runuser AzDevOps -c /bin/bash /agent/run.sh
root        4653    4650  0 13:24 ?        00:00:00 runuser AzDevOps -c /bin/bash /agent/run.sh
AzDevOps    4654    4653  0 13:24 ?        00:00:00 /bin/bash /agent/run.sh
AzDevOps    4658    4654  0 13:24 ?        00:00:05 /agent/bin/Agent.Listener run
vtrevam+    6856    6600  0 15:18 pts/0    00:00:00 grep --color=auto Az

vtrevamp-admin@vtrevamp-devops-buildagent0000TS:/agent/_work/_tasks$ ls -ltr /agent/
total 80
-rwxrwxr-x  1 AzDevOps AzDevOps  2014 Mar 19 13:27 run.sh
-rwxrwxr-x  1 AzDevOps AzDevOps  2753 Mar 19 13:27 run-docker.sh
-rwxrwxr-x  1 AzDevOps AzDevOps  3170 Mar 19 13:27 reauth.sh
-rwxrwxr-x  1 AzDevOps AzDevOps  9465 Mar 19 13:27 license.html
-rwxrwxr-x  1 AzDevOps AzDevOps   726 Mar 19 13:27 env.sh
-rwxrwxr-x  1 AzDevOps AzDevOps  3173 Mar 19 13:27 config.sh
drwxrwxr-x  7 AzDevOps AzDevOps  4096 Mar 19 13:29 externals
drwxrwxr-x 26 AzDevOps AzDevOps 20480 Mar 19 13:39 bin
-rwxrwxr-x  1 AzDevOps AzDevOps  5539 Apr 10 13:23 enableagent.sh
-rwxr-xr-x  1 AzDevOps AzDevOps  4671 Apr 10 13:24 svc.sh
drwxr-xr-x 11 AzDevOps AzDevOps  4096 Apr 10 13:29 _work
drwxr-xr-x  3 AzDevOps AzDevOps  4096 Apr 10 14:33 _diag

vtrevamp-admin@vtrevamp-devops-buildagent0000TS:/agent/_work/_tasks$ ls -ltr /agent/_work
total 36
drwxr-xr-x 2 AzDevOps AzDevOps 4096 Apr 10 13:24 _tool
drwxr-xr-x 3 AzDevOps AzDevOps 4096 Apr 10 13:24 SourceRootMapping
drwxr-xr-x 6 AzDevOps AzDevOps 4096 Apr 10 13:24 1
drwxr-xr-x 2 root     root     4096 Apr 10 13:24 _tasks
drwxr-xr-x 6 AzDevOps AzDevOps 4096 Apr 10 13:25 2
drwxr-xr-x 2 AzDevOps AzDevOps 4096 Apr 10 13:25 _temp
drwxr-xr-x 6 AzDevOps AzDevOps 4096 Apr 10 13:25 3
drwxr-xr-x 6 AzDevOps AzDevOps 4096 Apr 10 14:32 4
drwxr-xr-x 6 AzDevOps AzDevOps 4096 Apr 10 14:33 5

vtrevamp-admin@vtrevamp-devops-buildagent0000TS:/agent/_work/_tasks$ ls -ltr /agent/_work/_tasks

[dieter-adesso]

Some additional notes and findings:

1. It seems, that it only happens directly after an agent-machine has been launched by Scale-Set and hasn't processed any jobs before (almost 100% sure).

2. It also happens only for Deploy-Pipelines having an Approval-Gate in a separate Job, but it doesn't happen each time, just about 10% of all executions.

3. We configured:

• "Maximum number of virtual machines in the scale set" = 2
• "Number of agents to keep on standby" = 0
• "Delay in minutes before deleting excess idle agents" = 30

4. In Diagnostics we see a a max # of 11-18 jobs queued over time.

5. Whenever the error with Access to path exists, this agent cannot be used for any further Job that uses Tasks, because any subsequent job on this agent hits this error during "Initialize Job" phase. The agent must be stopped first. Afterwards the same failed job can proceed and finish normally.

6. Since the folder is owned by root, it cannot be caused by just wrong Pipeline programming. We don't have privileges to chown as root.

More details can be demanded via pm. Meanwhile, we collected a lot of information.

[dieter-adesso]

I've isolated the issue and can reproduce it by 100%.
It is a clear Azure DevOps Bug!

Precondition to hit this bug:

• Declarative deployment pipeline with Approval-Job
• Agent hosted on Ubuntu 22.04 LTS
• Az-DevOps Agent is hosted on a VM-ScaleSet
• Pipeline with deployment splitted into two jobs (Job A is just for approval, Job B shall deploy).
• Pipeline waits on Approval Gate and while approving, there should not be any agent running.

When the new Agent starts, it gets initialized to run the first job of any given pipeline. It must be assured, that the Pipeline with the Approval will be the first one to be processed on a freshly started agent.
Since there is no real task or step to be processed, the initialization seems to be incomplete and results in a folder like:
drwxr-xr-x 2 root root 4096 Apr 10 13:24 /agent/_work/_tasks

Since it is owned by root instead of AzDevOps, any subsequent job will fail.

As workaround, I've added a dummy-step into the Approval-job "just a tiny bash with echo" is sufficient.
Additionally, you need a "dependsOn" in the next job, which was set implicitly if you don't have any task in the approval-job.

In this case, the initialization by the approval job will be done completely and any subsequent jobs will work properly.