Pool provider is sometimes slow

[Cheesebaron]

Description

I wrote in the developer community here: https://developercommunity.visualstudio.com/t/macos-13-Pool-provider-seems-slow/10522135#T-ND10523938 Tong Wu told me to log a bug here.

We are often seeing this message in Azure Pipelines that pool provider is slow and often builds are stuck waiting for macOS-13 agents for upwards of 40 minutes. This is even though in our concurrent builds of 25 there are 20+ agents idling.

You can find some of the affected builds here:
https://trackman.visualstudio.com/Golf/_build/results?buildId=395103&view=results
https://trackman.visualstudio.com/Golf/_build/results?buildId=395072&view=results

It doesn't happen every time, but it happens several times on different pipelines during the day.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

Agent name: 'Azure Pipelines 2'
Agent machine name: 'Mac-1700593125567'
Current agent version: '3.227.2'
Operating System
macOS
13.6
22G120
Runner Image
Image: macos-13
Version: 20231025.2
Included Software: https://github.com/actions/runner-images/blob/macos-13/20231025.2/images/macos/macos-13-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macos-13%2F20231025.2
Runner Image Provisioner
2.0.312.1
Current image version: '20231025.2'

Is it regression?

yes

Expected behavior

Not having to wait upwards of 40 minutes for agent to become available

Actual behavior

Pool provider is slow and there is a long wait for agents

Repro steps

Just queue a macOS-13 vmpool build and observe

[vpolikarpov-akvelon]

Hey @Cheesebaron. Thank you for reaching out, but there is nothing to do on the side of the runner images generation team. Please contact GitHub Support.

[Cheesebaron]

Nice to be sent from place to place to get support.

Can GitHub support help me with Azure Pipelines?

[unrateddi]

@Cheesebaron we are facing the same issue here and changed to macos-13-arm64 which seems to work fine. So, maybe you could try that, if that image works for your needs, until they resolve the issue.

[Cheesebaron]

@Cheesebaron we are facing the same issue here and changed to macos-13-arm64 which seems to work fine. So, maybe you could try that, if that image works for your needs, until they resolve the issue.

These don't seem to be available in Azure DevOps pipelines

[unrateddi]

@Cheesebaron that's weird. I have successfully run Azure Pipelines with that image. I even tried to run it now in case something changed.

https://github.com/actions/runner-images/blob/macos-13/20231025.2/images/macos/macos-13-arm64-Readme.md

[Cheesebaron]

Maybe your organization has access to a preview of some sorts?

[unrateddi]

@Cheesebaron hmm maybe, unfortunately I can't know it for sure.

[qwertylolman]

we are facing the same issue. since we were forced to use macos-13 to build xamarin apps and we also were forced to migrate to net7. furthermore, we cannot even build the flutter apps, because of xcode lack of support for macos-12. this issue is quite old, I started to face it maybe 2-3 months ago. when I started the migration to net7.

[mmrazik]

Please note that the only supported images on Azure DevOps are the following: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#software

When it comes to MacOS 13 the only vm image available for Azure DevOps is macOS-13 (in beta).

Anything else is strongly discouraged and it may stop working for you without prior notice.

[qwertylolman]

Please note that the only supported images on Azure DevOps are the following: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#software

When it comes to MacOS 13 the only vm image available for Azure DevOps is macOS-13 (in beta).

yeah, it's quite confusing that arm64 is not listed anywhere, but presented in the repository xD
https://github.com/actions/runner-images/blob/main/images/macos/macos-13-arm64-Readme.md

but, seems like this image doesn't have the issue. i've switched our flutter project. now it's building a bit faster and without delays. xamarin net7 projects, on the other hand, are having some strange freezes during the dotnet build step, we are currently investigating this as for now.