Skip to content

CI: run the GPU build legs on demand only - #7

Merged
danielhanchen merged 1 commit into
masterfrom
ci/gpu-legs-manual
Aug 8, 2026
Merged

CI: run the GPU build legs on demand only#7
danielhanchen merged 1 commit into
masterfrom
ci/gpu-legs-manual

Conversation

@danielhanchen

Copy link
Copy Markdown
Member

The GPU legs in build.yml build nothing we ship, and they are 85% of the CI bill. This gates them to workflow_dispatch so they stay runnable on demand without running on every push and PR.

Nothing here ships

Our prebuilts come from unsloth-sd-prebuilt.yml, which is CPU and Apple only by design. Its own header says why:

# and Apple (Metal). GPU hosts use diffusers/torch, so no CUDA/ROCm/Vulkan here.

Its matrix is five labels: Darwin-macOS-arm64, Darwin-macOS-x86_64, Linux-Ubuntu-22.04-x86_64, Linux-Ubuntu-24.04-aarch64, win-cpu-x64. I checked every release in this repo: no CUDA, ROCm or Vulkan asset has ever been published.

build.yml is upstream leejet's CI, inherited by the fork, and its own release job has never published here either. It fails on every push:

##[error]Resource not accessible by integration

Consistent across the run history (2026-08-08T04:09 push failure, 2026-08-08T02:24 push failure, ...).

Nor does the forced-native GPU path use them. studio/install_sd_cpp_prebuilt.py:43-44 falls back to leejet/stable-diffusion.cpp when our mirror cannot serve a host, so --accelerator rocm pulls upstream's asset, not ours.

What they cost

Per-job wall time on run 31238801839:

minutes job
GPU 156.8 windows-latest-rocm
GPU 146.1 ubuntu-latest-rocm
GPU 132.9 windows-latest-cmake (cuda12)
GPU 70.0 container images (cuda, linux/arm64)
GPU 17.8 container images (cuda, linux/amd64)
GPU 17.3 ubuntu-latest-cmake-vulkan
GPU 11.1 container images (vulkan)
GPU 8.9 windows-latest-cmake (vulkan)
cpu 13.6 ubuntu-latest-cmake
cpu 9.6 macOS-latest-cmake
cpu 5.8 windows-latest-cmake (cpu)

561 of 658 minutes, 85%. They are also the source of the multi-GB artifacts that were filling Actions storage (sd-cudart-...cu12-x64.zip at 3.14 GiB, ...win-cuda12-x64.zip at 2.02 GiB, plus the ROCm zips), none of it published.

What still runs on push and PR

ubuntu-latest-cmake, macOS-latest-cmake and windows-latest-cmake (cpu), about 29 minutes. That keeps the reason to have upstream CI on a fork at all: an early signal that a sync has broken the compile. What it stops paying for is compiling backends we do not ship.

One thing to look at closely

Matrix entries cannot carry a job-level if:, so windows-latest-cmake uses a fromJSON ternary. It is one long line and it is the least pretty part of this change. I verified it mechanically rather than by eye: parsed both branches out of the expression and compared them to the original entries.

dispatch branch == original matrix : True
push branch      == cpu entry only : True
cpu defines byte-identical         : True
cuda12 defines byte-identical      : True
identical outside gates+matrix     : True

If you would rather not carry that line, the alternative is to move defines out of the matrix into a lookup inside the build step so the matrix holds only build names. That is a cleaner file but a larger diff into the steps, so I did not do it unasked.

Side effect

release lists the gated jobs in needs, so on a master push it now skips instead of failing. Since it has never succeeded in this repo, that removes a standing red X rather than losing a capability. If you want build.yml releases to actually work, that is a separate fix to its token permissions, and this PR would then be hiding the failure instead of resolving it. Worth deciding deliberately.

Each gate is one line, so reverting any single leg restores its old behaviour.

Stacked on top of #6, which fixes the ccache settings for these same ROCm legs so an on-demand run is not needlessly slow.

Nothing this workflow builds ships. Our prebuilts come from
unsloth-sd-prebuilt.yml, which is CPU and Apple only by design because GPU
hosts run diffusers/torch. No release here has ever carried a CUDA, ROCm or
Vulkan asset, and build.yml's own release job fails on every push with
"Resource not accessible by integration", so it has never published anything.
Even a forced native GPU load does not use these: install_sd_cpp_prebuilt.py
falls back to leejet upstream when our mirror cannot serve the host.

Measured on run 31238801839, the GPU legs are 561 of 658 minutes, 85% of the
run, and they are the source of the multi-GB artifacts that were filling
Actions storage.

Gated to workflow_dispatch: ubuntu-latest-cmake-vulkan, windows-latest-rocm,
ubuntu-latest-rocm, the container images job, and the cuda12 and vulkan entries
of windows-latest-cmake. What still runs on push and PR is the compile smoke
signal worth keeping: ubuntu-latest-cmake, macOS-latest-cmake and windows CPU,
about 29 minutes.

The matrix entries cannot take a job-level if, so that one uses a fromJSON
ternary. Verified both branches parse back to the original entries with the
defines byte-identical, and that the file is otherwise unchanged outside the
five gates.

Side effect worth knowing: release needs the gated jobs, so on a master push it
now skips instead of failing. It has never succeeded here, so this removes a
standing red X rather than losing a capability. Reverting any single gate
restores the old behaviour for that leg.
@danielhanchen
danielhanchen merged commit 22e2879 into master Aug 8, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant