[macOS] macos-15 (x86) runner is excruciatingly slow

[dumbmoron]

Description

I have recently noticed that the x86 macos-15 runner has become uselessly slow. I have no idea whether it's overloaded, or if it's a regression, but it is incapable of compiling software at a normal rate.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 22.04
• Ubuntu 24.04
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022
• Windows Server 2025

Image version and build link

Example job: https://github.com/imputnet/helium-macos/actions/runs/16128303494/job/45562831983

Current runner version: '2.325.0'
Runner Image Provisioner
Hosted Compute Agent
Version: 20250703.357
Commit: 07daf62238a21140d93e045a38f3784d75c509e1
Build Date: 2025-07-03T14:39:09Z
Operating System
macOS
15.5
24F74
Runner Image
Image: macos-15-arm64
Version: 20250630.1914
Included Software: https://github.com/actions/runner-images/blob/macos-15-arm64/20250630.1914/images/macos/macos-15-arm64-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macos-15-arm64%2F20250630.1914

Is it regression?

Maybe

Expected behavior

The runner should be able to compile more than 400 translation units over 5 hours

Actual behavior

The runner is incapable of compiling more than 400 units into objects over 5 hours. Previously, it was able to compile thousands (>6000) objects per hour.

Repro steps

1. Try to compile a larger project
2. Fail spectacularly

[ijunaidm1]

Hi @dumbmoron - We will start investigating on this issue and will update you accordingly.

[amlcurran]

Can confirm that

Image: macos-15-arm64
Version: 20250630.1914

Is also running much slower for us than previous, similar to my previously reported issues #11509

[sangeeths03]

Hi @dumbmoron, thanks for the detailed report.

We've tested this thoroughly on our side using a synthetic workload that compiles over 6000 translation units into a single executable. This benchmark was executed across macos-13-arm64, macos-14-arm64, and macos-15-arm64 runners using the latest image version , and we did not observe any significant slowdown on macos-15-arm64 compared to earlier versions.

System conditions also appear normal:

1. ~35 GiB of free disk space (df -h)

2. Build times are within expected ranges

3. Full run log: - https://github.com/sangeeths03/runner-images-sangeeth/actions/runs/16216838686

Given these results, we’re currently unable to reproduce the performance issue on our side.

As a next step, we'll fork your repository and attempt to reproduce the problem using your exact build and configuration. This should help us identify whether the slowdown is related to specific build scripts, link step behavior, or other project-specific factors.

Additionally, a new macOS 15 ARM runner image is scheduled for release this week. Once it's live, we recommend trying your build again to see if the issue still persists under the updated environment.

We'll follow up after deeper testing with your repo.
Also, could you please confirm:

1. What build duration you were observing previously for this same job (before the slowdown)?
2. If the slowness is consistent or intermittent across reruns?

Thanks again for your patience as we investigate this further.

[dumbmoron]

Hi @sangeeths03,

Thank you for looking into this!

What build duration you were observing previously for this same job (before the slowdown)?

The build normally takes around 20 hours to complete (4x 5 hour build jobs). Recently, I noticed that it can take easily up to 50 hours (10x 5 hour build jobs).

If the slowness is consistent or intermittent across reruns?

The slowness does seem to be intermittent - during some builds, the runner builds at a normal pace (maybe 1 object file per second, more or less, or much faster if it's using ccached objects). In other builds, the pipeline slows to a crawl, producing 1 object file per 10 to 20 minutes.

We currently have a x86 pipeline experiencing this: https://github.com/imputnet/helium-macos/actions/runs/16207265303/job/45787787964
It has been running for 2 hours so far, and only managed to produce 259 object files (normally, it is able to compile around 1200-1600 units per hour).