Git commit
master-758-c674225
Operating System & Version
Debian 13
GGML backends
CUDA
Command-line arguments used
sd-cli -p "a cat" --backend "diffusion=cuda0&cuda1" --diffusion-model z_image_turbo-Q5_0.gguf --vae flux1vae.safetensors --llm Z-Image-AbliteratedV1.Q6_K.gguf
Steps to reproduce
- Build stable-diffusion.cpp with CUDA support (version master-758-c674225).
- Run inference using the z_image_turbo model across 2 GPUs using the following command:
./sd-cli -p "a cat" --backend "diffusion=cuda0&cuda1" --diffusion-model z_image_turbo-Q5_0.gguf --vae flux1vae.safetensors --llm Z-Image-AbliteratedV1.Q6_K.gguf
What you expected to happen
The model should run successfully on multiple GPUs without crashing, just like the flux1 model does on the exact same multi-GPU configuration and backend setup. Also, the z_image_turbo model works completely fine when running on a single GPU, so it is expected to work in multi-GPU mode as well.
What actually happened
The application crashes right after loading the tensors with a CUDA error on the second device (device 1). Here is the log output:
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 19.39s (read: 18.91s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.30s)
[ERROR] ggml_extend.hpp:70 - CUDA error: an illegal memory access was encountered
[ERROR] ggml_extend.hpp:70 - current device: 1, in function ggml_cuda_kernel_can_use_pdl at common.cuh:1627
[ERROR] ggml_extend.hpp:70 - cudaFuncGetAttributes(&attr, kernel) ggml-cuda.cu:104: CUDA error
Logs / error messages / stack trace
iam@ai:~/AI/SD$ ./sd-cli -p "a cat" --backend "diffusion=cuda0&cuda1" --diffusion-model z_image_turbo-Q5_0.gguf --vae flux1vae.safetensors --llm Z-Image-AbliteratedV1.Q6_K.gguf
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 19941 MiB):
Device 0: NVIDIA CMP 50HX, compute capability 7.5, VMM: yes, VRAM: 9970 MiB
Device 1: NVIDIA CMP 50HX, compute capability 7.5, VMM: yes, VRAM: 9970 MiB
[INFO ] stable-diffusion.cpp:630 - loading diffusion model from 'z_image_turbo-Q5_0.gguf'
[INFO ] model_loader.cpp:236 - load z_image_turbo-Q5_0.gguf using gguf format
[INFO ] stable-diffusion.cpp:692 - loading llm from 'Z-Image-AbliteratedV1.Q6_K.gguf'
[INFO ] model_loader.cpp:236 - load Z-Image-AbliteratedV1.Q6_K.gguf using gguf format
[INFO ] stable-diffusion.cpp:706 - loading vae from 'flux1vae.safetensors'
[INFO ] model_loader.cpp:239 - load flux1vae.safetensors using safetensors format
[INFO ] stable-diffusion.cpp:757 - Version: Z-Image
[INFO ] stable-diffusion.cpp:808 - Weight type stat: f32: 640 | q5_0: 180 | q8_0: 22 | q6_K: 253
[INFO ] stable-diffusion.cpp:809 - Conditioner weight type stat: f32: 145 | q6_K: 253
[INFO ] stable-diffusion.cpp:810 - Diffusion model weight type stat: f32: 251 | q5_0: 180 | q8_0: 22
[INFO ] stable-diffusion.cpp:811 - VAE weight type stat: f32: 244
[INFO ] layer_split_partition.cpp:210 - Diffusion model layer split: CUDA0 <- blocks [0, 11) + non-block tensors
[INFO ] layer_split_partition.cpp:210 - Diffusion model layer split: CUDA1 <- blocks [11, 30)
[INFO ] stable-diffusion.cpp:1247 - using VAE for encoding / decoding
[INFO ] auto_encoder_kl.hpp:527 - vae decoder: ch = 128
[INFO ] stable-diffusion.cpp:1481 - total params memory size = 8818.96MB (VRAM 8818.96MB, RAM 0.00MB): text_encoders 4326.88MB(VRAM), diffusion_model 4332.08MB(VRAM), vae
160.00MB(VRAM), controlnet 0.00MB(N/A), extensions 0.00MB(N/A)
[INFO ] stable-diffusion.cpp:1590 - running in FLOW mode
[INFO ] stable-diffusion.cpp:4964 - generate_image 512x512
[INFO ] denoiser.hpp:1026 - get_sigmas with discrete scheduler
[INFO ] stable-diffusion.cpp:3861 - sampling using Euler method
|##################################################| 386/386 - 107.84MB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 28.45s (read: 25.81s, memcpy: 0.00s, convert: 0.06s, copy_to_backend: 0.21s)
[INFO ] stable-diffusion.cpp:4662 - get_learned_condition completed, taking 28.63s
[INFO ] stable-diffusion.cpp:5010 - generating image: 1/1 - seed 42
|##################################################| 452/452 - 223.43MB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 19.39s (read: 18.91s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.30s)
[ERROR] ggml_extend.hpp:70 - CUDA error: an illegal memory access was encountered
[ERROR] ggml_extend.hpp:70 - current device: 1, in function ggml_cuda_kernel_can_use_pdl at common.cuh:1627
[ERROR] ggml_extend.hpp:70 - cudaFuncGetAttributes(&attr, kernel) ggml-cuda.cu:104: CUDA error
[New LWP 33509]
[New LWP 33508]
[New LWP 33507]
[New LWP 33506]
[New LWP 33499]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007faa86ca69ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#0 0x00007faa86ca69ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007faa86c9b668 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007faa86c9b6ad in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007faa86d067c7 in wait4 () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000564d495cabeb in ggml_print_backtrace ()
#5 0x0000564d495cad38 in ggml_abort ()
#6 0x0000564d489cb2c3 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) ()
#7 0x0000564d48b19786 in quantize_row_q8_1_cuda(float const*, int const*, void*, ggml_type, long, long, long, long, long, long, long, long, CUstream_st*) ()
#8 0x0000564d48b0d9f2 in ggml_cuda_mul_mat_vec_q(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor const*, ggml_tensor*, ggml_cuda_mm_fusio
n_args_host const*) ()
#9 0x0000564d489d2d51 in ggml_cuda_try_fuse(ggml_backend_cuda_context*, ggml_cgraph*, int) ()
#10 0x0000564d489e2207 in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) ()
#11 0x0000564d495eb8e1 in ggml_backend_sched_compute_splits(ggml_backend_sched*) ()
#12 0x0000564d495ec5bb in ggml_backend_sched_graph_compute ()
#13 0x0000564d4881e267 in std::optional<sd::Tensor > GGMLRunner::execute_graph(ggml_cgraph*, int, bool, bool, bool, bool, std::unordered_set<std::__cxx11::b
asic_string<char, std::char_traits, std::allocator >, std::hash<std::_cxx11::basic_string<char, std::char_traits, std::allocator > >, std::equal
to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocat
or > > > const*) ()
#14 0x0000564d48692bed in std::optional<sd::Tensor > GGMLRunner::compute(std::function<ggml_cgraph* ()>, int, bool, bool, bool, bool) [clone .constprop.0] (
)
#15 0x0000564d4883b2df in ZImage::ZImageRunner::compute(int, DiffusionParams const&) ()
#16 0x0000564d486c085a in StableDiffusionGGML::sample(std::shared_ptr const&, bool, sd::Tensor const&, sd::Tensor, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator > const&, std::vector<sd::Tensor, std::allocator<sd::Tensor > > const&, bool, sd::Tensor const&, sd::Tensor const&, float, int, float, sd_cache_params_t const*, sd::Tensor const&)::{lambda(sd::Tensor const&, float, int)#1}::operator()(sd::Tensor const&, float, int) const::{lambda(SDCondition const&, sd::Tensor const*, std::vector<int, std::allocator > const*, std::vector<sd::Tensor, std::allocator<sd::Tensor > > const*)#1}::operator()(SDCondition const&, sd::Tensor const*, std::vector<int, std::allocator > const*, std::vector<sd::Tensor, std::allocator<sd::Tensor > > const*) const ()
#17 0x0000564d48835dbc in StableDiffusionGGML::sample(std::shared_ptr const&, bool, sd::Tensor const&, sd::Tensor, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator > const&, std::vector<sd::Tensor, std::allocator<sd::Tensor > > const&, bool, sd::Tensor const&, sd::Tensor const&, float, int, float, sd_cache_params_t const*, sd::Tensor const&)::{lambda(sd::Tensor const&, float, int)#1}::operator()(sd::Tensor const&, float, int) const ()
#18 0x0000564d48839186 in std::_Function_handler<sd::guidance::GuiderOutput (sd::Tensor const&, float, int), StableDiffusionGGML::sample(std::shared_ptr const&, bool, sd::Tensor const&, sd::Tensor, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator > const&, std::vector<sd::Tensor, std::allocator<sd::Tensor > > const&, bool, sd::Tensor const&, sd::Tensor const&, float, int, float, sd_cache_params_t const*, sd::Tensor const&)::{lambda(sd::Tensor const&, float, int)#1}>::_M_invoke(std::_Any_data const&, sd::Tensor const&, float&&, int&&) ()
#19 0x0000564d487a68da in StableDiffusionGGML::sample(std::shared_ptr const&, bool, sd::Tensor const&, sd::Tensor, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator > const&, std::vector<sd::Tensor, std::allocator<sd::Tensor > > const&, bool, sd::Tensor const&, sd::Tensor const&, float, int, float, sd_cache_params_t const*, sd::Tensor const&) ()
#20 0x0000564d48697ab3 in generate_image ()
#21 0x0000564d484d3e63 in main ()
[Inferior 1 (process 33498) detached]
Additional context / environment details
For reference, here is the successful log output when running the flux1 model on the exact same multi-GPU setup:
sd-cli -p "a cat" --backend "diffusion=cuda0&cuda1" --diffusion-model genImg/Flux/flux1-dev-Q5_K_S.gguf --vae VAE/flux1vae.safetensors --t5xxl Clip/t5-v1_1-xxl-encoder-Q5_K_S.gguf --clip_l Clip/clip_l.safetensors
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 19941 MiB):
Device 0: NVIDIA CMP 50HX, compute capability 7.5, VMM: yes, VRAM: 9970 MiB
Device 1: NVIDIA CMP 50HX, compute capability 7.5, VMM: yes, VRAM: 9970 MiB
[INFO ] stable-diffusion.cpp:630 - loading diffusion model from 'genImg/Flux/flux1-dev-Q5_K_S.gguf'
[INFO ] model_loader.cpp:236 - load genImg/Flux/flux1-dev-Q5_K_S.gguf using gguf format
[INFO ] stable-diffusion.cpp:653 - loading clip_l from 'Clip/clip_l.safetensors'
[INFO ] model_loader.cpp:239 - load Clip/clip_l.safetensors using safetensors format
[INFO ] stable-diffusion.cpp:677 - loading t5xxl from 'Clip/t5-v1_1-xxl-encoder-Q5_K_S.gguf'
[INFO ] model_loader.cpp:236 - load Clip/t5-v1_1-xxl-encoder-Q5_K_S.gguf using gguf format
[INFO ] stable-diffusion.cpp:706 - loading vae from 'VAE/flux1vae.safetensors'
[INFO ] model_loader.cpp:239 - load VAE/flux1vae.safetensors using safetensors format
[INFO ] stable-diffusion.cpp:757 - Version: Flux
[INFO ] stable-diffusion.cpp:808 - Weight type stat: f32: 765 | f16: 201 | q5_K: 472 | q6_K: 1
[INFO ] stable-diffusion.cpp:809 - Conditioner weight type stat: f32: 50 | f16: 196 | q5_K: 168 | q6_K: 1
[INFO ] stable-diffusion.cpp:810 - Diffusion model weight type stat: f32: 471 | f16: 5 | q5_K: 304
[INFO ] stable-diffusion.cpp:811 - VAE weight type stat: f32: 244
[INFO ] layer_split_partition.cpp:210 - Diffusion model layer split: CUDA0 <- blocks [0, 12) + non-block tensors
[INFO ] layer_split_partition.cpp:210 - Diffusion model layer split: CUDA1 <- blocks [12, 38)
[INFO ] stable-diffusion.cpp:1247 - using VAE for encoding / decoding
[INFO ] auto_encoder_kl.hpp:527 - vae decoder: ch = 128
[INFO ] stable-diffusion.cpp:1481 - total params memory size = 11835.23MB (VRAM 11835.23MB, RAM 0.00MB): text_encoders 3773.84MB(VRAM), diffusion_model 7901.39MB(VRAM), v
ae 160.00MB(VRAM), controlnet 0.00MB(N/A), extensions 0.00MB(N/A)
[INFO ] stable-diffusion.cpp:1596 - running in Flux FLOW mode
[INFO ] stable-diffusion.cpp:4964 - generate_image 512x512
[INFO ] denoiser.hpp:1088 - get_sigmas with Flux scheduler
[INFO ] stable-diffusion.cpp:3861 - sampling using Euler method
Git commit
master-758-c674225
Operating System & Version
Debian 13
GGML backends
CUDA
Command-line arguments used
sd-cli -p "a cat" --backend "diffusion=cuda0&cuda1" --diffusion-model z_image_turbo-Q5_0.gguf --vae flux1vae.safetensors --llm Z-Image-AbliteratedV1.Q6_K.gguf
Steps to reproduce
./sd-cli -p "a cat" --backend "diffusion=cuda0&cuda1" --diffusion-model z_image_turbo-Q5_0.gguf --vae flux1vae.safetensors --llm Z-Image-AbliteratedV1.Q6_K.gguf
What you expected to happen
The model should run successfully on multiple GPUs without crashing, just like the flux1 model does on the exact same multi-GPU configuration and backend setup. Also, the z_image_turbo model works completely fine when running on a single GPU, so it is expected to work in multi-GPU mode as well.
What actually happened
The application crashes right after loading the tensors with a CUDA error on the second device (device 1). Here is the log output:
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 19.39s (read: 18.91s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.30s)
[ERROR] ggml_extend.hpp:70 - CUDA error: an illegal memory access was encountered
[ERROR] ggml_extend.hpp:70 - current device: 1, in function ggml_cuda_kernel_can_use_pdl at common.cuh:1627
[ERROR] ggml_extend.hpp:70 - cudaFuncGetAttributes(&attr, kernel) ggml-cuda.cu:104: CUDA error
Logs / error messages / stack trace
iam@ai:~/AI/SD$ ./sd-cli -p "a cat" --backend "diffusion=cuda0&cuda1" --diffusion-model z_image_turbo-Q5_0.gguf --vae flux1vae.safetensors --llm Z-Image-AbliteratedV1.Q6_K.gguf
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 19941 MiB):
Device 0: NVIDIA CMP 50HX, compute capability 7.5, VMM: yes, VRAM: 9970 MiB
Device 1: NVIDIA CMP 50HX, compute capability 7.5, VMM: yes, VRAM: 9970 MiB
[INFO ] stable-diffusion.cpp:630 - loading diffusion model from 'z_image_turbo-Q5_0.gguf'
[INFO ] model_loader.cpp:236 - load z_image_turbo-Q5_0.gguf using gguf format
[INFO ] stable-diffusion.cpp:692 - loading llm from 'Z-Image-AbliteratedV1.Q6_K.gguf'
[INFO ] model_loader.cpp:236 - load Z-Image-AbliteratedV1.Q6_K.gguf using gguf format
[INFO ] stable-diffusion.cpp:706 - loading vae from 'flux1vae.safetensors'
[INFO ] model_loader.cpp:239 - load flux1vae.safetensors using safetensors format
[INFO ] stable-diffusion.cpp:757 - Version: Z-Image
[INFO ] stable-diffusion.cpp:808 - Weight type stat: f32: 640 | q5_0: 180 | q8_0: 22 | q6_K: 253
[INFO ] stable-diffusion.cpp:809 - Conditioner weight type stat: f32: 145 | q6_K: 253
[INFO ] stable-diffusion.cpp:810 - Diffusion model weight type stat: f32: 251 | q5_0: 180 | q8_0: 22
[INFO ] stable-diffusion.cpp:811 - VAE weight type stat: f32: 244
[INFO ] layer_split_partition.cpp:210 - Diffusion model layer split: CUDA0 <- blocks [0, 11) + non-block tensors
[INFO ] layer_split_partition.cpp:210 - Diffusion model layer split: CUDA1 <- blocks [11, 30)
[INFO ] stable-diffusion.cpp:1247 - using VAE for encoding / decoding
[INFO ] auto_encoder_kl.hpp:527 - vae decoder: ch = 128
[INFO ] stable-diffusion.cpp:1481 - total params memory size = 8818.96MB (VRAM 8818.96MB, RAM 0.00MB): text_encoders 4326.88MB(VRAM), diffusion_model 4332.08MB(VRAM), vae
160.00MB(VRAM), controlnet 0.00MB(N/A), extensions 0.00MB(N/A)
[INFO ] stable-diffusion.cpp:1590 - running in FLOW mode
[INFO ] stable-diffusion.cpp:4964 - generate_image 512x512
[INFO ] denoiser.hpp:1026 - get_sigmas with discrete scheduler
[INFO ] stable-diffusion.cpp:3861 - sampling using Euler method
|##################################################| 386/386 - 107.84MB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 28.45s (read: 25.81s, memcpy: 0.00s, convert: 0.06s, copy_to_backend: 0.21s)
[INFO ] stable-diffusion.cpp:4662 - get_learned_condition completed, taking 28.63s
[INFO ] stable-diffusion.cpp:5010 - generating image: 1/1 - seed 42
|##################################################| 452/452 - 223.43MB/s
[INFO ] model_loader.cpp:1247 - loading tensors completed, taking 19.39s (read: 18.91s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 0.30s)
[ERROR] ggml_extend.hpp:70 - CUDA error: an illegal memory access was encountered
[ERROR] ggml_extend.hpp:70 - current device: 1, in function ggml_cuda_kernel_can_use_pdl at common.cuh:1627
[ERROR] ggml_extend.hpp:70 - cudaFuncGetAttributes(&attr, kernel) ggml-cuda.cu:104: CUDA error
[New LWP 33509]
[New LWP 33508]
[New LWP 33507]
[New LWP 33506]
[New LWP 33499]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007faa86ca69ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#0 0x00007faa86ca69ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007faa86c9b668 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007faa86c9b6ad in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007faa86d067c7 in wait4 () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000564d495cabeb in ggml_print_backtrace ()
#5 0x0000564d495cad38 in ggml_abort ()
#6 0x0000564d489cb2c3 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) ()
#7 0x0000564d48b19786 in quantize_row_q8_1_cuda(float const*, int const*, void*, ggml_type, long, long, long, long, long, long, long, long, CUstream_st*) ()
#8 0x0000564d48b0d9f2 in ggml_cuda_mul_mat_vec_q(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor const*, ggml_tensor*, ggml_cuda_mm_fusio
n_args_host const*) ()
#9 0x0000564d489d2d51 in ggml_cuda_try_fuse(ggml_backend_cuda_context*, ggml_cgraph*, int) ()
#10 0x0000564d489e2207 in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) ()
#11 0x0000564d495eb8e1 in ggml_backend_sched_compute_splits(ggml_backend_sched*) ()
#12 0x0000564d495ec5bb in ggml_backend_sched_graph_compute ()
#13 0x0000564d4881e267 in std::optional<sd::Tensor > GGMLRunner::execute_graph(ggml_cgraph*, int, bool, bool, bool, bool, std::unordered_set<std::__cxx11::b
asic_string<char, std::char_traits, std::allocator >, std::hash<std::_cxx11::basic_string<char, std::char_traits, std::allocator > >, std::equal
to<std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocat
or > > > const*) ()
#14 0x0000564d48692bed in std::optional<sd::Tensor > GGMLRunner::compute(std::function<ggml_cgraph* ()>, int, bool, bool, bool, bool) [clone .constprop.0] (
)
#15 0x0000564d4883b2df in ZImage::ZImageRunner::compute(int, DiffusionParams const&) ()
#16 0x0000564d486c085a in StableDiffusionGGML::sample(std::shared_ptr const&, bool, sd::Tensor const&, sd::Tensor, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator > const&, std::vector<sd::Tensor, std::allocator<sd::Tensor > > const&, bool, sd::Tensor const&, sd::Tensor const&, float, int, float, sd_cache_params_t const*, sd::Tensor const&)::{lambda(sd::Tensor const&, float, int)#1}::operator()(sd::Tensor const&, float, int) const::{lambda(SDCondition const&, sd::Tensor const*, std::vector<int, std::allocator > const*, std::vector<sd::Tensor, std::allocator<sd::Tensor > > const*)#1}::operator()(SDCondition const&, sd::Tensor const*, std::vector<int, std::allocator > const*, std::vector<sd::Tensor, std::allocator<sd::Tensor > > const*) const ()
#17 0x0000564d48835dbc in StableDiffusionGGML::sample(std::shared_ptr const&, bool, sd::Tensor const&, sd::Tensor, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator > const&, std::vector<sd::Tensor, std::allocator<sd::Tensor > > const&, bool, sd::Tensor const&, sd::Tensor const&, float, int, float, sd_cache_params_t const*, sd::Tensor const&)::{lambda(sd::Tensor const&, float, int)#1}::operator()(sd::Tensor const&, float, int) const ()
#18 0x0000564d48839186 in std::_Function_handler<sd::guidance::GuiderOutput (sd::Tensor const&, float, int), StableDiffusionGGML::sample(std::shared_ptr const&, bool, sd::Tensor const&, sd::Tensor, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator > const&, std::vector<sd::Tensor, std::allocator<sd::Tensor > > const&, bool, sd::Tensor const&, sd::Tensor const&, float, int, float, sd_cache_params_t const*, sd::Tensor const&)::{lambda(sd::Tensor const&, float, int)#1}>::_M_invoke(std::_Any_data const&, sd::Tensor const&, float&&, int&&) ()
#19 0x0000564d487a68da in StableDiffusionGGML::sample(std::shared_ptr const&, bool, sd::Tensor const&, sd::Tensor, SDCondition const&, SDCondition const&, SDCondition const&, sd::Tensor const&, float, sd_guidance_params_t const&, float, int, sample_method_t, bool, char const*, std::vector<float, std::allocator > const&, std::vector<sd::Tensor, std::allocator<sd::Tensor > > const&, bool, sd::Tensor const&, sd::Tensor const&, float, int, float, sd_cache_params_t const*, sd::Tensor const&) ()
#20 0x0000564d48697ab3 in generate_image ()
#21 0x0000564d484d3e63 in main ()
[Inferior 1 (process 33498) detached]
Additional context / environment details
For reference, here is the successful log output when running the flux1 model on the exact same multi-GPU setup:
sd-cli -p "a cat" --backend "diffusion=cuda0&cuda1" --diffusion-model genImg/Flux/flux1-dev-Q5_K_S.gguf --vae VAE/flux1vae.safetensors --t5xxl Clip/t5-v1_1-xxl-encoder-Q5_K_S.gguf --clip_l Clip/clip_l.safetensors
ggml_cuda_init: found 2 CUDA devices (Total VRAM: 19941 MiB):
Device 0: NVIDIA CMP 50HX, compute capability 7.5, VMM: yes, VRAM: 9970 MiB
Device 1: NVIDIA CMP 50HX, compute capability 7.5, VMM: yes, VRAM: 9970 MiB
[INFO ] stable-diffusion.cpp:630 - loading diffusion model from 'genImg/Flux/flux1-dev-Q5_K_S.gguf'
[INFO ] model_loader.cpp:236 - load genImg/Flux/flux1-dev-Q5_K_S.gguf using gguf format
[INFO ] stable-diffusion.cpp:653 - loading clip_l from 'Clip/clip_l.safetensors'
[INFO ] model_loader.cpp:239 - load Clip/clip_l.safetensors using safetensors format
[INFO ] stable-diffusion.cpp:677 - loading t5xxl from 'Clip/t5-v1_1-xxl-encoder-Q5_K_S.gguf'
[INFO ] model_loader.cpp:236 - load Clip/t5-v1_1-xxl-encoder-Q5_K_S.gguf using gguf format
[INFO ] stable-diffusion.cpp:706 - loading vae from 'VAE/flux1vae.safetensors'
[INFO ] model_loader.cpp:239 - load VAE/flux1vae.safetensors using safetensors format
[INFO ] stable-diffusion.cpp:757 - Version: Flux
[INFO ] stable-diffusion.cpp:808 - Weight type stat: f32: 765 | f16: 201 | q5_K: 472 | q6_K: 1
[INFO ] stable-diffusion.cpp:809 - Conditioner weight type stat: f32: 50 | f16: 196 | q5_K: 168 | q6_K: 1
[INFO ] stable-diffusion.cpp:810 - Diffusion model weight type stat: f32: 471 | f16: 5 | q5_K: 304
[INFO ] stable-diffusion.cpp:811 - VAE weight type stat: f32: 244
[INFO ] layer_split_partition.cpp:210 - Diffusion model layer split: CUDA0 <- blocks [0, 12) + non-block tensors
[INFO ] layer_split_partition.cpp:210 - Diffusion model layer split: CUDA1 <- blocks [12, 38)
[INFO ] stable-diffusion.cpp:1247 - using VAE for encoding / decoding
[INFO ] auto_encoder_kl.hpp:527 - vae decoder: ch = 128
[INFO ] stable-diffusion.cpp:1481 - total params memory size = 11835.23MB (VRAM 11835.23MB, RAM 0.00MB): text_encoders 3773.84MB(VRAM), diffusion_model 7901.39MB(VRAM), v
ae 160.00MB(VRAM), controlnet 0.00MB(N/A), extensions 0.00MB(N/A)
[INFO ] stable-diffusion.cpp:1596 - running in Flux FLOW mode
[INFO ] stable-diffusion.cpp:4964 - generate_image 512x512
[INFO ] denoiser.hpp:1088 - get_sigmas with Flux scheduler
[INFO ] stable-diffusion.cpp:3861 - sampling using Euler method