feat: support ggml backend select and enhance the build system by Cyberhan123 · Pull Request #1368 · leejet/stable-diffusion.cpp

Cyberhan123 · 2026-03-26T05:25:55Z

This pull request makes significant improvements to the CI build workflows across Linux, macOS, and Windows, with a strong focus on modernizing ROCm (AMD GPU) support, optimizing build caching, and unifying build configurations. The changes streamline GPU build matrix handling, update ROCm installation and packaging, and introduce ccache for faster builds.

Important change at : ggml_extend_backend.hpp

Key changes include:

ROCm/AMD GPU Build Modernization:

Replaces the legacy Windows HIP build with a new windows-latest-rocm job using ROCm 7.11.0, updating installation to use prebuilt tarballs and adjusting environment variables and packaging accordingly.
Updates the Linux ROCm job to support both ROCm 7.2 (legacy, via apt) and ROCm 7.11.0 (via tarball), with a matrix for GPU targets and ROCm versions, and modernizes the build and packaging steps.
Removes manual copying of ROCm runtime libraries in the Linux workflow, simplifying the packaging process.

Build Configuration and Matrix Unification:

Refactors the build matrix for CPU, CUDA, and Vulkan builds to use unified and modern CMake flags, replacing legacy AVX/AVX2/AVX512 variants with a single "cpu" variant and updating CUDA/Vulkan flags for consistency.
Updates all build steps to use more explicit and comprehensive CMake options for backend selection, RPATH handling, and parallel builds.

Build Performance and Maintenance:

Adds ccache to all major build jobs (Linux, macOS, Windows) to speed up rebuilds and reduce CI times.
Updates artifact naming to include ROCm version, improving traceability of build outputs.

General Improvements:

Adds the cmake/ directory to CMAKE_MODULE_PATH in CMakeLists.txt to support custom CMake modules.
Cleans up and removes unnecessary steps, such as free disk space operations and legacy AVX512 checks.

These changes make the build system more robust, future-proof, and maintainable, especially for AMD GPU support and multi-platform builds.

This PR Requires merging with previous PRs :
#1184,
#1281,
#1282

This will resolve the potential problems with the following PRs:
#1259

fix sdxl conditionner backends fix sd3 backend display

…end support, and eliminate hard-coded backend references.

Copilot

Pull request overview

This PR adds runtime GGML backend device selection (including multi-backend setups and RPC devices) and refactors the CMake build/install packaging to better support these new backend workflows.

Changes:

Introduces named-backend initialization helpers and new C API utilities to list available devices and register RPC devices.
Extends sd_ctx_params_t to allow selecting devices per model component (diffusion/CLIP/VAE/ControlNet/etc.) and updates model/component construction accordingly.
Restructures the CMake build into subdirectories and adds install artifacts (pkg-config + CMake package config), plus updates CLI/server docs and flags.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
src/z_image.hpp	Switches backend-specific numeric workarounds from compile-time macros to runtime backend detection.
src/qwen_image.hpp	Same as above (runtime Vulkan precision workaround).
src/common_block.hpp	Same as above (runtime Vulkan precision workaround).
src/util.h	Adds `sd_backend_is()` declaration and includes ggml backend header.
src/util.cpp	Implements `sd_backend_is()`.
src/ggml_extend.hpp	Adds named-backend init helpers; extends LoRA adapter interfaces to be backend-aware; updates cast/sync helpers.
src/lora.hpp	Passes backend through LoRA weight-diff/cast paths so Vulkan-specific casting can be applied correctly.
src/conditioner.hpp	Enables multiple backends for multi-encoder conditioners; adds per-encoder backend accessors.
src/stable-diffusion.cpp	Implements per-component backend initialization, RPC device registration, and backend/device listing APIs; refactors backend usage throughout initialization and LoRA flows.
src/upscaler.cpp	Adds backend device selection for upscaler initialization via named backend.
src/model.cpp	Removes backend-specific includes (moving toward backend-agnostic compilation).
include/stable-diffusion.h	Extends `sd_ctx_params_t` with per-component device strings; extends upscaler API; adds device listing + RPC registration APIs.
examples/common/common.hpp	Adds CLI/server flags and wiring for per-component backend devices; adjusts option parsing flow control.
examples/cli/main.cpp	Adds `--rpc` and `--list-devices`; passes selected upscaler backend device into the upscaler API.
examples/cli/README.md	Documents new CLI flags and backend device selection options.
examples/server/README.md	Documents new server flags and backend device selection options; removes CPU-only component toggles.
examples/CMakeLists.txt	Adjusts examples build flags/includes and subdirectories.
docs/rpc.md	New documentation for building/using RPC server setups with this project.
src/CMakeLists.txt	New src-level library target definition.
cmake/common.cmake	New shared compile-flag helpers (warnings/sanitizers).
cmake/build-info.cmake	New build metadata extraction (git commit/count, compiler, target).
cmake/sd.pc.in	Adds pkg-config template for installation.
cmake/sd-config.cmake.in	Adds CMake package config template for installation.
CMakeLists.txt	Major build refactor: module path, shared/static defaults, ggml option transitions, add `src/` subdir build, and install/package generation.

Comments suppressed due to low confidence (1)

CMakeLists.txt:160

The if(GGML_SYCL) block calls target_compile_options(${SD_LIB} ...) before ${SD_LIB} is created (the target is defined later in src/CMakeLists.txt). This will fail configuration when GGML_SYCL=ON. Move the SYCL-specific target_compile_options() call to after add_subdirectory(src) (or apply the flags via variables/target properties inside src/CMakeLists.txt).

# Is this needed?
if(GGML_SYCL)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-narrowing -fsycl")
    # disable fast-math on host, see:
    # https://www.intel.com/content/www/us/en/docs/cpp-compiler/developer-guide-reference/2021-10/fp-model-fp.html
    if (WIN32)
        set(SYCL_COMPILE_OPTIONS /fp:precise)
    else()
        set(SYCL_COMPILE_OPTIONS -fp-model=precise)
    endif()
    message("-- Turn off fast-math for host in SYCL backend")
    target_compile_options(${SD_LIB} PRIVATE ${SYCL_COMPILE_OPTIONS})
endif()

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/util.cpp

src/stable-diffusion.cpp

examples/cli/main.cpp

src/stable-diffusion.cpp

cmake/sd-config.cmake.in

src/conditioner.hpp

src/ggml_extend.hpp

This builds from ROCm release tarballs. In order to use it you must have ROCm installed in the operating system first. - Update GPU_TARGETS to match Linux job (7.11.0 targets) - Cache DLL's from extracted tarball directory instead of installer directory - Align Windows and Linux ROCm build configurations Also; rename the legacy ROCm task to clarify which artifacts come from which release.

Copilot

Pull request overview

Copilot reviewed 35 out of 40 changed files in this pull request and generated 13 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/stable-diffusion.cpp

CMakeLists.txt

src/CMakeLists.txt

.github/workflows/build.yml

src/stable-diffusion.cpp

Dockerfile.vulkan

.github/workflows/build.yml

docs/rpc.md

Cyberhan123 · 2026-03-28T14:21:30Z

I think this PR is ready for review and can be merged.
cc @stduhpf @leejet @Green-Sky

stduhpf · 2026-03-28T14:33:10Z

src/auto_encoder_kl.hpp

-                 1.8029f, 1.7786f, 1.7868f, 1.7837f, 1.7717f, 1.7590f, 1.7610f, 1.7479f,
-                 1.7336f, 1.7373f, 1.7340f, 1.7343f, 1.8626f, 1.8527f, 1.8629f, 1.8589f,
-                 1.7593f, 1.7526f, 1.7556f, 1.7583f, 1.7363f, 1.7400f, 1.7355f, 1.7394f,
-                 1.7342f, 1.7246f, 1.7392f, 1.7304f, 1.7551f, 1.7513f, 1.7559f, 1.7488f,
-                 1.8449f, 1.8454f, 1.8550f, 1.8535f, 1.8240f, 1.7813f, 1.7854f, 1.7945f,
-                 1.8047f, 1.7876f, 1.7695f, 1.7676f, 1.7782f, 1.7667f, 1.7925f, 1.7848f,
-                 1.7579f, 1.7407f, 1.7483f, 1.7368f, 1.7961f, 1.7998f, 1.7920f, 1.7925f,
-                 1.7780f, 1.7747f, 1.7727f, 1.7749f, 1.7526f, 1.7447f, 1.7657f, 1.7495f,
-                 1.7775f, 1.7720f, 1.7813f, 1.7813f, 1.8162f, 1.8013f, 1.8023f, 1.8033f,
-                 1.7527f, 1.7331f, 1.7563f, 1.7482f, 1.7610f, 1.7507f, 1.7681f, 1.7613f,
-                 1.7665f, 1.7545f, 1.7828f, 1.7726f, 1.7896f, 1.7999f, 1.7864f, 1.7760f,
-                 1.7613f, 1.7625f, 1.7560f, 1.7577f, 1.7783f, 1.7671f, 1.7810f, 1.7799f,
-                 1.7201f, 1.7068f, 1.7265f, 1.7091f, 1.7793f, 1.7578f, 1.7502f, 1.7455f,
-                 1.7587f, 1.7500f, 1.7525f, 1.7362f, 1.7616f, 1.7572f, 1.7444f, 1.7430f,
-                 1.7509f, 1.7610f, 1.7634f, 1.7612f, 1.7254f, 1.7135f, 1.7321f, 1.7226f,
-                 1.7664f, 1.7624f, 1.7718f, 1.7664f, 1.7457f, 1.7441f, 1.7569f, 1.7530f};
+                1.8029f, 1.7786f, 1.7868f, 1.7837f, 1.7717f, 1.7590f, 1.7610f, 1.7479f,
+                1.7336f, 1.7373f, 1.7340f, 1.7343f, 1.8626f, 1.8527f, 1.8629f, 1.8589f,
+                1.7593f, 1.7526f, 1.7556f, 1.7583f, 1.7363f, 1.7400f, 1.7355f, 1.7394f,
+                1.7342f, 1.7246f, 1.7392f, 1.7304f, 1.7551f, 1.7513f, 1.7559f, 1.7488f,
+                1.8449f, 1.8454f, 1.8550f, 1.8535f, 1.8240f, 1.7813f, 1.7854f, 1.7945f,
+                1.8047f, 1.7876f, 1.7695f, 1.7676f, 1.7782f, 1.7667f, 1.7925f, 1.7848f,
+                1.7579f, 1.7407f, 1.7483f, 1.7368f, 1.7961f, 1.7998f, 1.7920f, 1.7925f,
+                1.7780f, 1.7747f, 1.7727f, 1.7749f, 1.7526f, 1.7447f, 1.7657f, 1.7495f,
+                1.7775f, 1.7720f, 1.7813f, 1.7813f, 1.8162f, 1.8013f, 1.8023f, 1.8033f,
+                1.7527f, 1.7331f, 1.7563f, 1.7482f, 1.7610f, 1.7507f, 1.7681f, 1.7613f,
+                1.7665f, 1.7545f, 1.7828f, 1.7726f, 1.7896f, 1.7999f, 1.7864f, 1.7760f,
+                1.7613f, 1.7625f, 1.7560f, 1.7577f, 1.7783f, 1.7671f, 1.7810f, 1.7799f,
+                1.7201f, 1.7068f, 1.7265f, 1.7091f, 1.7793f, 1.7578f, 1.7502f, 1.7455f,
+                1.7587f, 1.7500f, 1.7525f, 1.7362f, 1.7616f, 1.7572f, 1.7444f, 1.7430f,
+                1.7509f, 1.7610f, 1.7634f, 1.7612f, 1.7254f, 1.7135f, 1.7321f, 1.7226f,
+                1.7664f, 1.7624f, 1.7718f, 1.7664f, 1.7457f, 1.7441f, 1.7569f, 1.7530f};


This should probably be reverted

This uses the format-code.ps1 script. I haven't found any problems with the script...

stduhpf · 2026-03-28T14:41:44Z

Looks good at first glance, but I'm not sure having the build system changes AND the backend functionality in a single PR is a good Idea. It makes reviewing harder, and it will also make troubleshooting more complicated in the future once it's squashed and merged. The backend selection + RPC support was already a big piece.

Cyberhan123 · 2026-03-28T14:45:40Z

Looks good at first glance, but I'm not sure having the build system changes AND the backend functionality in a single PR is a good Idea. It makes reviewing harder, and it will also make troubleshooting more complicated in the future once it's squashed and merged. The backend selection + RPC support was already a big piece.

This was a last resort. I applied your and @superm1 patches mainly to solve backend build issues, so I suggested the following order in the PR: merge your and @superm1 's PR into the main branch first. This way, my codebase will be much smaller.

Cyberhan123 · 2026-03-28T14:49:18Z

@stduhpf Your current code branch has some potential issues, and I suggest you don't make any changes, as I've already fixed them. The best solution is to merge all our code together.

stduhpf · 2026-03-28T14:51:28Z

Maybe you could point to the commit(s) that fixed the issues, so I can cherry-pick them, and hopefully this would make it compatible with your branch.

Cyberhan123 · 2026-03-28T14:55:39Z

Maybe you could point to the commit(s) that fixed the issues, so I can cherry-pick them, and hopefully this would make it compatible with your branch.

I think this is the most important fix; it seems your code isn't initializing vision_backend.

e4d143b

stduhpf · 2026-03-28T15:05:16Z

I will try to make sure your branch can still seemlessly rebase on mine

stduhpf and others added 24 commits March 26, 2026 10:20

Select backend devices via arg

b0545ab

fix build

f89620c

show backend device description

61af83e

CLI: add --list-devices arg

c8ffbd3

null-terminate even if buffer is too small

4e7bdb3

move stuff to ggml_extend.cpp

53d32a9

--upscaler-backend-device

1800c9a

use diffusion_backend for loading LoRAs

c1fc286

--photomaker-backend-device (+fixes)

9de5032

--vision-backend-device

a85121f

check backends at runtime

4d0b24f

fix missing includes

ead7116

fix typo

4597013

multiple clip backend devices

10763be

fix sdxl conditionner backends fix sd3 backend display

update help message

f3076ea

Add RPC documentation

a97a978

update docs

20e6ba4

update RPC docs

970304e

fix apply_loras_immediately when using different non-CPU backends

a792579

Force sequencial tensor loading when using RPC

9e3229b

fix build

61ed023

Get first stage backend for loading loras

8fe2ccc

Fix lora loading when using multiple clip backends

ae5b8ab

rm all head code for ggml

229664d

Copilot AI review requested due to automatic review settings March 26, 2026 05:25

Copilot started reviewing on behalf of Cyberhan123 March 26, 2026 05:26 View session

feat: enhance the build system by adding CMake configuration and back…

e302ee5

…end support, and eliminate hard-coded backend references.

Copilot AI reviewed Mar 26, 2026

View reviewed changes

Cyberhan123 added 2 commits March 26, 2026 16:10

fix review

83f7bab

format code

1556395

Cyberhan123 added 2 commits March 28, 2026 11:47

try fix ci

3e07a2b

add ggml-cpu.h

54f2762

loci-dev mentioned this pull request Mar 28, 2026

UPSTREAM PR #1368: feat: support ggml backend select and enhance the build system auroralabs-loci/stable-diffusion.cpp#94

Open

Cyberhan123 and others added 9 commits March 28, 2026 16:58

fix use cpu backend

17db074

fix sycl docker build

7b5d14f

Update HIP target to use 26.Q1 (ROCm 7.1.1)

dbea191

update rocm new build

f246d83

sync and update rocm build

29576a3

fix and add ccache

ef54ba4

fix rocm build

1e23dd4

fix rocm cache path

5ff332b

Cyberhan123 requested a review from Copilot March 28, 2026 13:22

Copilot started reviewing on behalf of Cyberhan123 March 28, 2026 13:22 View session

add ninja_job

9f4fc8d

Copilot AI reviewed Mar 28, 2026

View reviewed changes

Cyberhan123 added 3 commits March 28, 2026 21:31

rm -j params

355dcda

fix cr

ca6c96d

format code

1a31a23

Cyberhan123 added 2 commits March 28, 2026 22:21

fix version backend

e4d143b

format code

8c8f2d0

stduhpf reviewed Mar 28, 2026

View reviewed changes

fix hard_code with SD_USE_VULKAN

0527541

Conversation

Cyberhan123 commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Cyberhan123 commented Mar 28, 2026

Uh oh!

stduhpf Mar 28, 2026

Choose a reason for hiding this comment

Uh oh!

Cyberhan123 Mar 28, 2026

Choose a reason for hiding this comment

Uh oh!

stduhpf commented Mar 28, 2026

Uh oh!

Cyberhan123 commented Mar 28, 2026

Uh oh!

Cyberhan123 commented Mar 28, 2026

Uh oh!

stduhpf commented Mar 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Cyberhan123 commented Mar 28, 2026

Uh oh!

stduhpf commented Mar 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Cyberhan123 commented Mar 26, 2026 •

edited

Loading

stduhpf commented Mar 28, 2026 •

edited

Loading