ci : migrate ggml ci to self-hosted runners

[ggerganov]

• With this change, there is no longer need to specify ggml-ci into commits to trigger the CI
• The V100 runner now runs both CUDA and Vulkan CI
• Add T4 runner for CUDA, Vulkan Coopmat1 and Vulkan Coopmat2 runs
• The Mac Mini runner will run both Metal and Vulkan CI
• Disable test-backend-ops from Debug builds due to slowness (alternatively, we can add a Release build config with asserts enabled and run just that)
• Add instructions for adding self-hosted runners
• Move the MUSA CI instructions to separate README
• Add AMD V710 runner for Vulkan and ROCm workflows
• Switch to more lightweight Qwen3 0.6B model

[netrunnereve]

It'll definitely be good to have a Vulkan CI. Is it not crashing now on Nvidia?

BTW the V100 is pretty old and I wonder if it's worth upgrading to something newer with coopmat and coopmat2 support. Budget wise it should be possible to move the existing arm and x86 runs to the free Github machines (I think 16GB and 4 cores are enough for the CI) or have the amx machine do double duty as a x86 CPU runner.

[ggerganov]

It'll definitely be good to have a Vulkan CI. Is it not crashing now on Nvidia?

On both V100 and T4, the F16 EXP tests are failing because instead of inf the Vulkan F16 kernels return 65504.000000:

[EXP] inf mismatch: Vulkan0=65504.000000 CPU=inf   EXP(type=f16,ne_a=[128,2,2,2],v=0): FAIL
[EXP] inf mismatch: Vulkan0=65504.000000 CPU=inf   EXP(type=f16,ne_a=[5,7,11,13],v=0): FAIL

Is this a known issue and can we workaround?

[am17an]

Maybe #15652?

[ggerganov]

Maybe #15652?

Yes, probably a patch like this is needed. I'll leave it to be fixed on master.

@netrunnereve

For the Vulkan runs, we now have:

• V100 (no coopmat)
• T4 with coopmat1
• T4 with coopmat2
• Will add a Mac workflow on Tuesday

Let me know if there are any additional Vulkan configs that we can add. Or any machines from the Azure cloud that can be useful to exercise in addition to the current ones.

[netrunnereve]

Let me know if there are any additional Vulkan configs that we can add. Or any machines from the Azure cloud that can be useful to exercise in addition to the current ones.

For Vulkan your runs already cover most cases including the FP16 path (Mac with MoltenVK), the int dot path (V100), and coopmat 1 and 2 (T4). We also already run the tests for the FP16 path with llvmpipe using the standard Github machines. The only remaining path is the FP32 one (run with GGML_VK_DISABLE_F16, GGML_VK_DISABLE_COOPMAT, GGML_VK_DISABLE_COOPMAT2, and GGML_VK_DISABLE_INTEGER_DOT_PRODUCT) which is only used on old gpus. If we have room for that then go for it, else I don't think it's that important.

If you're thinking of getting another machine then the NVads V710 is an AMD one which could be used for the ROCM ci as well as an additional Vulkan testcase for benchmarking and detecting driver bugs. Having an Intel GPU would be nice for SYCL and Vulkan as well but Azure doesn't have those.

FYI while I was looking into this I also discovered that the V100 Azure machines will be discontinued at the end of this month so that'll have to be dealt with eventually.

[slaren]

I think this is a good starting point. Some notes:

• A lot of this could be consolidated into a single job using a matrix
• It may be too heavy to run the full ggml-ci on every PR push, it may be necessary to leave the full version for master only, and use lighter tests for PRs

[ggerganov]

@netrunnereve Do you know which driver I need to install for the AMD V710 GPU?

https://github.com/ggml-org/llama.cpp/actions/runs/17892383263/job/50874519651?pr=16116

Currently, I have installed the Vulkan SDK and the project builds. But it does not detect the GPU.

Edit: I think I found it: https://learn.microsoft.com/en-us/azure/virtual-machines/linux/azure-n-series-amd-gpu-driver-linux-installation-guide

[CISC]

* It may be too heavy to run the full ggml-ci on every PR push, it may be necessary to leave the full version for master only, and use lighter tests for PRs

Perhaps limit them to pushes to the related backend?