[BUG]: Slowed down builds on hosted agents

[Jens-Schmitt]

What happened?

Since 10.04.2024 our build no longer takes 25 minutes, but suddenly more than 60 minutes. After approx. 10 min => 30-40 build tasks the CPU is at 100% and remains at this value. Then all tasks are delayed until the pipeline timeout occurs after 2 hours.

No changes on the build definition yaml file, no code changes. Image type: windows-latest.

Versions

Current agent version: '3.236.1'
Operating System
Microsoft Windows Server 2022
10.0.20348
Datacenter
Current image version: '20240407.1.0'

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

No response

Version controll system

GIT

Relevant log output

No response

[DenisNikulin5]

Hi @Jens-Schmitt
Could you please provide the agent logs to get more details?

[KonstantinTyukalov]

Hi @Jens-Schmitt, could you please also share pipeline yaml slice to reproduce the issue?

[Jens-Schmitt]

Build-enc.m.yml.txt

[Jens-Schmitt]

@DenisNikulin5: The agent logs contain private information that I am not allowed to share. I could extract the task headers - if that would be enough...?

[KonstantinTyukalov]

@Jens-Schmitt could you please create a ticket in Developer Community to have a private communication channel?
Agent logs would really help us to investigate the root cause of the issue, so please share with us all the info which is possible in your case.
If you decide to send us the agent logs, please do so in a private conversation and ensure that all sensitive data is removed.
Thank you

[Jens-Schmitt]

You can find the logs here: https://developercommunity.visualstudio.com/t/Slowed-down-build-on-hosted-agent/10638569. If you need more information about the log files or additional logging should be included in the build definition, please let me know.

[Jens-Schmitt]

$(AZP_ENABLE_RESOURCE_UTILIZATION_WARNINGS) has apparently been removed today.

We are back to normal. The builds run again for 25 min. There is no longer a timeout after 2 h. The CPU no longer goes to 100%.

Thanks for the investigation and the effort.

[pratiksoni1408]

Hello @Jens-Schmitt ,

May I know what the root cause was as I am facing same issue.