Vulkan: is it possible to work around maxBufferSize?

[drHuangMHT]

Hi! Thanks in advance for your great work with sd and vulkan support!

I have a MI50 32GB GPU but despite the ample VRAM, I cannot run VAE decode on it:

System Info:
    SSE3 = 1
    AVX = 1
    AVX2 = 1
    AVX512 = 0
    AVX512_VBMI = 0
    AVX512_VNNI = 0
    FMA = 1
    NEON = 0
    ARM_FMA = 0
    F16C = 1
    FP16_VA = 0
    WASM_SIMD = 0
    VSX = 0
[DEBUG] stable-diffusion.cpp:174  - Using Vulkan backend
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Pro VII (AMD proprietary driver) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 32768 | matrix cores: none
// model info omitted
[INFO ] stable-diffusion.cpp:244  - Version: SDXL
[INFO ] stable-diffusion.cpp:277  - Weight type:                 f32
[INFO ] stable-diffusion.cpp:278  - Conditioner weight type:     f32
[INFO ] stable-diffusion.cpp:279  - Diffusion model weight type: f32
[INFO ] stable-diffusion.cpp:280  - VAE weight type:             f32
[DEBUG] stable-diffusion.cpp:282  - ggml tensor size = 400 bytes
// FP16 VAE warning on SDXL omitted
[DEBUG] ggml_extend.hpp:1174 - clip params backend buffer size =  469.44 MB(VRAM) (196 tensors)
[DEBUG] ggml_extend.hpp:1174 - clip params backend buffer size =  2649.92 MB(VRAM) (517 tensors)
[DEBUG] ggml_extend.hpp:1174 - unet params backend buffer size =  9099.29 MB(VRAM) (1680 tensors)
[DEBUG] ggml_extend.hpp:1174 - vae params backend buffer size =  94.47 MB(VRAM) (140 tensors)
[DEBUG] stable-diffusion.cpp:419  - loading weights
[DEBUG] model.cpp:1727 - loading tensors
  |==================================================| 2641/2641 - 200.00it/s
[INFO ] stable-diffusion.cpp:503  - total params memory size = 12313.11MB (VRAM 12313.11MB, RAM 0.00MB): clip 3119.36MB(VRAM), unet 9099.29MB(VRAM), vae 94.47MB(VRAM), controlnet 0.00MB(VRAM), pmid 0.00MB(VRAM)
[INFO ] stable-diffusion.cpp:522  - loading model from 'D:\hassakuXLIllustrious_v2.safetensors' completed, taking 13.68s
[INFO ] stable-diffusion.cpp:556  - running in eps-prediction mode
[DEBUG] stable-diffusion.cpp:600  - finished loaded file
// logs related to image generation omitted
[DEBUG] ggml_extend.hpp:1126 - unet compute buffer size: 830.86 MB(VRAM)
  |==================================================| 20/20 - 3.95s/it
[INFO ] stable-diffusion.cpp:1478 - sampling completed, taking 80.33s
[INFO ] stable-diffusion.cpp:1486 - generating 1 latent images completed, taking 80.59s
[INFO ] stable-diffusion.cpp:1489 - decoding 1 latents
ggml_vulkan: Device memory allocation of size 6979321856 failed.
ggml_vulkan: Requested buffer size exceeds device memory allocation limit: ErrorOutOfDeviceMemory
ggml_gallocr_reserve_n: failed to allocate Vulkan0 buffer of size 6979321856 // ~6GB
[ERROR] ggml_extend.hpp:1119 - vae: failed to allocate the compute buffer

I believe it is caused by maxBufferSize being 2 147 483 648(~2GB out of 32GB total) in core 1.3 property of the GPU. The same applies to unet compute buffer. For reference, my Titan V reports maxBufferSize being 1 099 511 627 776(~10GB out of 12GB total).

However, using DirectML, all VRAM can be successfully allocated(at least that's what showed up on my Task Manager) with similar generation speed.

From what I can tell, the possibility of solving the issue on software level is very small, but I'm asking anyways because there is definitely someone who understand it better and answer it once and for all.

[stduhpf]

You can set the GGML_VK_FORCE_MAX_ALLOCATION_SIZE env variable to whatever value if you want to try bypass that limitation. But if you set it too high, the program might crash while initializing the Vulkan backend. On my machine, I can go up to 4 294 967 295 (the default is also 2 147 483 648 for me) without any issue.

As far as I know there is no easy way to completely avoid running into this limit with larger images.

Also, using --vae-tiling will significantly reduce the VAE buffer size (at the cost of slightly reducing the quality, in theory at least). Alternatively, --vae-on-cpu can be a decent option if you have a fast enough CPU and don't want to use tiling. Using TAESD is also an option, it takes little VRAM and is much faster than VAE, but the quality is significantly worse.

[drHuangMHT]

You can set the GGML_VK_FORCE_MAX_ALLOCATION_SIZE env variable to whatever value if you want to try bypass that limitation. But if you set it too high, the program might crash while initializing the Vulkan backend. On my machine, I can go up to 4 294 967 295 (the default is also 2 147 483 648 for me) without any issue.

I saw that on a reddit post and unfortunately it doesn't work for me, probably because I'm using Windows, or the GPU is too old(GCN5.1). Also it looks like a 32-bit integer. I don't know how nvidia cards' can be that large.

As far as I know there is no easy way to completely avoid running into this limit with larger images.

I think so but I have an easy ™️ and dumb solution: stitch multiple buffers together. And, since the unet fits, why wouldn't the image buffer? But again I have absolutely no knowledge about how Vulkan or GPU programming works.

Also, using --vae-tiling will significantly reduce the VAE buffer size (at the cost of slightly reducing the quality, in theory at least). Alternatively, --vae-on-cpu can be a decent option if you have a fast enough CPU and don't want to use tiling.

Yeah, sending VAE decoding to CPU works, that's what I did. I will give VAE tiling a try. But the unet compute buffer(what do you call it? latent?) is still too small compared to total VRAM available.

[stduhpf]

I think so but I have an easy ™️ and dumb solution: stitch multiple buffers together. And, since the unet fits, why wouldn't the image buffer? But again I have absolutely no knowledge about how Vulkan or GPU programming works.

It sounds easy, but then you'd have to adapt all the shaders/kernels to work with these "stitched" buffers. Seems like a lot of work.

[idostyle]

Seems to be due to setting no_alloc to true when setting up the compute context here

stable-diffusion.cpp/ggml_extend.hpp

Line 1095 in 10c6501

params.no_alloc = true;

and then manually allocating the whole required memory as one block in

stable-diffusion.cpp/ggml_extend.hpp

Line 1117 in 10c6501

if (!ggml_gallocr_reserve(compute_allocr, gf)) {

[wbruna]

@drHuangMHT , master-309-35843c7 includes a change in the ggml allocator that should help avoiding this issue in most cases. On my card, which has a 4G allocation limit:

[INFO ] stable-diffusion.cpp:2169 - decoding 1 latents
[DEBUG] ggml_extend.hpp:1550 - vae compute buffer size: 6696.21 MB(VRAM)
[DEBUG] stable-diffusion.cpp:1547 - computing vae decode graph completed, taking 13.59s
[INFO ] stable-diffusion.cpp:2179 - latent 1 decoded, taking 13.59s

[drHuangMHT]

@drHuangMHT , master-309-35843c7 includes a change in the ggml allocator that should help avoiding this issue in most cases. On my card, which has a 4G allocation limit

I cloned the latest master but I'm still unable to run vae decoding without tiling or high resolution image generation on the GPU, it still reports ErrorOutOfDeviceMemoey. Are you using a newer card(RDNA1 and above) or on Linux? I'm not really hoping sd to run well using Vulkan on this card anymore, maybe I should try Linux with ROCm(ROCm 6.2 still supports this card) instead.
For those who interested, this card runs LLM reasonably well.

[wbruna]

I cloned the latest master but I'm still unable to run vae decoding without tiling or high resolution image generation on the GPU, it still reports ErrorOutOfDeviceMemoey. Are you using a newer card(RDNA1 and above) or on Linux?

Yeah, RDNA3 with radv on Linux. I think that memory limit comes from radv: it's the same on my old iGPU (3400G).

I'm not really hoping sd to run well using Vulkan on this card anymore, maybe I should try Linux with ROCm(ROCm 6.2 still supports this card) instead. For those who interested, this card runs LLM reasonably well.

In my case, compared to ROCm, Vulkan is slower for generation steps (around 30%), but the VAE runs much faster (--vae-conv-direct works really well with radv); so I end up using Vulkan most of the time.

By the way, if you don't absolutely need to run everything on f32, f16 should have pretty much the same quality with much less peak VRAM usage (unlikely to help with the VAE, though).

[0cc4m]

@drHuangMHT Please post another log. It should at least not be trying to allocate 6GB anymore. But since the Windows driver still limits allocations to 2GB instead of 4GB like the Linux driver, you still have an issue. It may be possible to work around it.

[wbruna]

Doing some tests here with GGML_VK_FORCE_MAX_ALLOCATION_SIZE: SDXL at 1024x1024 needs 4608 MiB:

[INFO ] stable-diffusion.cpp:2169 - decoding 1 latents
[DEBUG] ggml_extend.hpp:62   - ggml_gallocr_reserve_n: reallocating Vulkan1 buffer from size 0.00 MiB to 8768.25 MiB
ggml_vulkan: Device memory allocation of size 4831838208 failed.
ggml_vulkan: Requested buffer size exceeds device memory allocation limit: ErrorOutOfDeviceMemory
[ERROR] ggml_extend.hpp:71   - ggml_gallocr_reserve_n: failed to allocate Vulkan1 buffer of size 9194176516
[ERROR] ggml_extend.hpp:1545 - vae: failed to allocate the compute buffer

With --vae-conv-direct the max allocation size is lowered to 1024 MiB:

[INFO ] stable-diffusion.cpp:2169 - decoding 1 latents
[DEBUG] ggml_extend.hpp:62   - ggml_gallocr_reserve_n: reallocating Vulkan1 buffer from size 0.00 MiB to 2944.25 MiB
ggml_vulkan: Device memory allocation of size 1073741824 failed.
ggml_vulkan: Requested buffer size exceeds device memory allocation limit: ErrorOutOfDeviceMemory
[ERROR] ggml_extend.hpp:71   - ggml_gallocr_reserve_n: failed to allocate Vulkan1 buffer of size 3087269892
[ERROR] ggml_extend.hpp:1545 - vae: failed to allocate the compute buffer

Tiling should also work much better in recent versions.

[drHuangMHT]

@drHuangMHT Please post another log. It should at least not be trying to allocate 6GB anymore. But since the Windows driver still limits allocations to 2GB instead of 4GB like the Linux driver, you still have an issue. It may be possible to work around it.

I did more tests and TL;DR:

resolution	vae tiling	vae-conv-direct	force alloc	uint64 mod1	unet alloc	vae alloc	meaningful image
1024x1024	❌	❌	❌	❌	830MB ✔️	2304MB ❌	❌
1024x1024	❌	✔️	❌	❌	✔️	2944MB ✔️	✔️
1536x1536	❌	✔️	❌	❌	3978MB ❌	❓	❌
1536x1536	❌	✔️	✔️	❌	3978MB ✔️	8496MB ❌	❌
1536x1536	❌	❌	✔️	✔️	✔️	21600MB ✔️	❌
1536x1536	❌	✔️	✔️	✔️	✔️	8496MB ✔️	✔️
1664x1664	❌	❌	✔️	✔️	5406MB ✔️	26406MB ✔️	❌
1664x1664	❌	✔️	✔️	✔️	✔️	8154MB ✔️	❌
1664x1664	✔️	❌	✔️	✔️	✔️	✔️	❌
1792x1792	❌	❌	✔️	✔️	7005MB ✔️	31556MB ⚠️ ❌	❌
1792x1792	❌	✔️	✔️	✔️	7005MB ✔️	10584MB ❌	❌
2048x2048	❌	❌	✔️	✔️	11254MB✔️	35654MB ⚠️ ❌	❌
2048x2048	❌	✔️	✔️	✔️	✔️	17153MB ❌	❌
2048x2048	✔️	❌	✔️	✔️	✔️	✔️	❌

¹: some modification made to ggml library, see below for details
^⚠️ : exceeding device memory

Here's the log from master-35843c7. I'm using MSYS2 UCRT64 to build the binary. Note that f32 and f16 takes the same amount of time and memory on this device.

$ ./sd.exe --height 1024 --width 1024 -m /d/hassakuXLIllustrious_v2.safetensors -p waterbottle --verbose
Option:
    n_threads:                         8
    mode:                              img_gen
    model_path:                        D:/hassakuXLIllustrious_v2.safetensors
    wtype:                             unspecified
    clip_l_path:
    clip_g_path:
    clip_vision_path:
    t5xxl_path:
    diffusion_model_path:
    high_noise_diffusion_model_path:
    vae_path:
    taesd_path:
    esrgan_path:
    control_net_path:
    embedding_dir:
    photo_maker_path:
    pm_id_images_dir:
    pm_id_embed_path:
    pm_style_strength:                 20.00
    output_path:                       output.png
    init_image_path:
    end_image_path:
    mask_image_path:
    control_image_path:
    ref_images_paths:
    control_video_path:
    increase_ref_index:                false
    offload_params_to_cpu:             false
    clip_on_cpu:                       false
    control_net_cpu:                   false
    vae_on_cpu:                        false
    diffusion flash attention:         false
    diffusion Conv2d direct:           false
    vae_conv_direct:                   false
    control_strength:                  0.90
    prompt:                            waterbottle
    negative_prompt:
    clip_skip:                         -1
    width:                             1024
    height:                            1024
    sample_params:                     (txt_cfg: 7.00, img_cfg: 7.00, distilled_guidance: 3.50, slg.layer_count: 3, slg.layer_start: 0.01, slg.layer_end:
0.20, slg.scale: 0.00, scheduler: default, sample_method: default, sample_steps: 20, eta: 0.00, shifted_timestep: 0)
    high_noise_sample_params:          (txt_cfg: 7.00, img_cfg: 7.00, distilled_guidance: 3.50, slg.layer_count: 3, slg.layer_start: 0.01, slg.layer_end:
0.20, slg.scale: 0.00, scheduler: default, sample_method: default, sample_steps: -1, eta: 0.00, shifted_timestep: 0)
    moe_boundary:                      0.875
    flow_shift:                        inf
    strength(img2img):                 0.75
    rng:                               cuda
    seed:                              42
    batch_count:                       1
    vae_tiling:                        false
    upscale_repeats:                   1
    chroma_use_dit_mask:               true
    chroma_use_t5_mask:                false
    chroma_t5_mask_pad:                1
    video_frames:                      1
    vace_strength:                     1.00
    fps:                               16
System Info:
    SSE3 = 1
    AVX = 1
    AVX2 = 1
    AVX512 = 0
    AVX512_VBMI = 0
    AVX512_VNNI = 0
    FMA = 1
    NEON = 0
    ARM_FMA = 0
    F16C = 1
    FP16_VA = 0
    WASM_SIMD = 0
    VSX = 0
[DEBUG] stable-diffusion.cpp:152  - Using Vulkan backend
[DEBUG] ggml_extend.hpp:62   - ggml_vulkan: Found 1 Vulkan devices:
[DEBUG] ggml_extend.hpp:62   - ggml_vulkan: 0 = AMD Radeon Pro VII (AMD proprietary driver) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory:
32768 | int dot: 1 | matrix cores: none
[INFO ] stable-diffusion.cpp:201  - loading model from 'D:/hassakuXLIllustrious_v2.safetensors'
[INFO ] model.cpp:1044 - load D:/hassakuXLIllustrious_v2.safetensors using safetensors format
[DEBUG] model.cpp:1151 - init from 'D:/hassakuXLIllustrious_v2.safetensors', prefix = ''
[INFO ] stable-diffusion.cpp:267  - Version: SDXL
[INFO ] stable-diffusion.cpp:298  - Weight type:                 f16
[INFO ] stable-diffusion.cpp:299  - Conditioner weight type:     f16
[INFO ] stable-diffusion.cpp:300  - Diffusion model weight type: f16
[INFO ] stable-diffusion.cpp:301  - VAE weight type:             f16
[DEBUG] stable-diffusion.cpp:303  - ggml tensor size = 400 bytes
[WARN ] stable-diffusion.cpp:308  - !!!It looks like you are using SDXL model. If you find that the generated images are completely black, try specifying
SDXL VAE FP16 Fix with the --vae parameter. You can find it here: https://huggingface.co/madebyollin/sdxl-vae-fp16-fix/blob/main/sdxl_vae.safetensors
[DEBUG] clip.hpp:171  - vocab size: 49408
[DEBUG] clip.hpp:182  - trigger word img already in vocab
[DEBUG] ggml_extend.hpp:1725 - clip params backend buffer size =  235.06 MB(VRAM) (196 tensors)
[DEBUG] ggml_extend.hpp:1725 - clip params backend buffer size =  1329.29 MB(VRAM) (517 tensors)
[DEBUG] ggml_extend.hpp:1725 - unet params backend buffer size =  4900.07 MB(VRAM) (1680 tensors)
[DEBUG] ggml_extend.hpp:1725 - vae params backend buffer size =  94.47 MB(VRAM) (140 tensors)
[DEBUG] stable-diffusion.cpp:565  - loading weights
[DEBUG] model.cpp:1961 - using 8 threads for model loading
[DEBUG] model.cpp:2044 - loading tensors from hassakuXLIllustrious_v2.safetensors
  |==================================================| 2641/2641 - 985.08it/s
[INFO ] model.cpp:2282 - loading tensors completed, taking 2.69s (process: 0.00s, read: 0.77s, memcpy: 0.00s, convert: 0.00s, copy_to_backend: 1.86s)
[INFO ] stable-diffusion.cpp:646  - total params memory size = 6558.89MB (VRAM 6558.89MB, RAM 0.00MB): text_encoders 1564.36MB(VRAM), diffusion_model 4900
.07MB(VRAM), vae 94.47MB(VRAM), controlnet 0.00MB(VRAM), pmid 0.00MB(VRAM)
[INFO ] stable-diffusion.cpp:714  - running in eps-prediction mode
[DEBUG] stable-diffusion.cpp:725  - finished loaded file
[DEBUG] stable-diffusion.cpp:2262 - generate_image 1024x1024
[INFO ] stable-diffusion.cpp:2383 - TXT2IMG
[INFO ] stable-diffusion.cpp:874  - attempting to apply 0 LoRAs
[INFO ] stable-diffusion.cpp:894  - apply_loras completed, taking 0.00s
[DEBUG] stable-diffusion.cpp:895  - prompt after extract and remove lora: "waterbottle"
[DEBUG] conditioner.hpp:345  - parse 'waterbottle' to [['waterbottle', 1], ]
[DEBUG] clip.hpp:311  - token length: 77
[DEBUG] ggml_extend.hpp:1550 - clip compute buffer size: 1.40 MB(VRAM)
[DEBUG] ggml_extend.hpp:1550 - clip compute buffer size: 2.33 MB(VRAM)
[DEBUG] ggml_extend.hpp:1550 - clip compute buffer size: 2.33 MB(VRAM)
[DEBUG] conditioner.hpp:479  - computing condition graph completed, taking 218 ms
[DEBUG] conditioner.hpp:345  - parse '' to [['', 1], ]
[DEBUG] clip.hpp:311  - token length: 77
[DEBUG] ggml_extend.hpp:1550 - clip compute buffer size: 1.40 MB(VRAM)
[DEBUG] ggml_extend.hpp:1550 - clip compute buffer size: 2.33 MB(VRAM)
[DEBUG] ggml_extend.hpp:1550 - clip compute buffer size: 2.33 MB(VRAM)
[DEBUG] conditioner.hpp:479  - computing condition graph completed, taking 191 ms
[INFO ] stable-diffusion.cpp:2049 - get_learned_condition completed, taking 412 ms
[INFO ] stable-diffusion.cpp:2072 - sampling using Euler A method
[INFO ] stable-diffusion.cpp:2121 - generating image: 1/1 - seed 42
[DEBUG] ggml_extend.hpp:1550 - unet compute buffer size: 830.86 MB(VRAM)
  |==================================================| 20/20 - 3.91s/it
[INFO ] stable-diffusion.cpp:2158 - sampling completed, taking 79.11s
[INFO ] stable-diffusion.cpp:2166 - generating 1 latent images completed, taking 79.18s
[INFO ] stable-diffusion.cpp:2169 - decoding 1 latents
ggml_vulkan: Device memory allocation of size 2_415_919_104 failed. // ~2GB, underscores manually added, same below
ggml_vulkan: Requested buffer size exceeds device memory allocation limit: ErrorOutOfDeviceMemory
[ERROR] ggml_extend.hpp:71   - ggml_gallocr_reserve_n: failed to allocate Vulkan0 buffer of size 9_194_176_516 // ~9GB for some reason
[ERROR] ggml_extend.hpp:1543 - vae: failed to allocate the compute buffer
Segmentation fault

by using --vae-conv-direct, the decoding is possible for resolution 1024x1024:

[DEBUG] ggml_extend.hpp:1550 - unet compute buffer size: 830.86 MB(VRAM)
  |==================================================| 20/20 - 3.90s/it
[INFO ] stable-diffusion.cpp:2158 - sampling completed, taking 78.90s
[INFO ] stable-diffusion.cpp:2166 - generating 1 latent images completed, taking 78.97s
[INFO ] stable-diffusion.cpp:2169 - decoding 1 latents
[DEBUG] ggml_extend.hpp:1550 - vae compute buffer size: 2944.25 MB(VRAM)
[DEBUG] stable-diffusion.cpp:1547 - computing vae decode graph completed, taking 2.51s
[INFO ] stable-diffusion.cpp:2179 - latent 1 decoded, taking 2.51s
[INFO ] stable-diffusion.cpp:2183 - decode_first_stage completed, taking 2.51s
[INFO ] stable-diffusion.cpp:2475 - generate_image completed in 81.89s
save result PNG image to 'output.png'

by setting GGML_VK_FORCE_MAX_ALLOCATION_SIZE to 4 294 967 295(unsigned long), generation at 1536x1536 is possible but vae decoding will fail even with --vae-conv-direct(without tiling):

[INFO ] stable-diffusion.cpp:2072 - sampling using Euler A method
[INFO ] stable-diffusion.cpp:2121 - generating image: 1/1 - seed 42
[DEBUG] ggml_extend.hpp:1550 - unet compute buffer size: 3978.83 MB(VRAM)
  |==================================================| 20/20 - 9.15s/it
[INFO ] stable-diffusion.cpp:2158 - sampling completed, taking 186.43s
[INFO ] stable-diffusion.cpp:2166 - generating 1 latent images completed, taking 186.50s
[INFO ] stable-diffusion.cpp:2169 - decoding 1 latents
ggml_vulkan: Device memory allocation of size 5435817984 failed.
ggml_vulkan: Requested buffer size exceeds device memory allocation limit: ErrorOutOfDeviceMemory
[ERROR] ggml_extend.hpp:71   - ggml_gallocr_reserve_n: failed to allocate Vulkan0 buffer of size 8909291524
[ERROR] ggml_extend.hpp:1543 - vae: failed to allocate the compute buffer
Segmentation fault

However, this is interesting: I changed the code that reads the environment variable here

        const char* GGML_VK_FORCE_MAX_ALLOCATION_SIZE = getenv("GGML_VK_FORCE_MAX_ALLOCATION_SIZE");

        if (GGML_VK_FORCE_MAX_ALLOCATION_SIZE != nullptr) {
            device->max_memory_allocation_size = std::stoul(GGML_VK_FORCE_MAX_ALLOCATION_SIZE); // 32-bit unsigned integer
        } 
        // unrelated branches omitted

to

// unchanged code omitted
device->max_memory_allocation_size = std::stoull(GGML_VK_FORCE_MAX_ALLOCATION_SIZE); // now 64-bit unsigned integer

and setting GGML_VK_FORCE_MAX_ALLOCATION_SIZE to 9_511_627_776 which is around 9GB, the allocation succeeded(9_511_627_776 > 8_909_291_524) for some reason and the decoding produced an normal image!

As for even larger allocation size, I tried 2048x2048 with conv-direct, it produced a different error:

// after setting GGML_VK_FORCE_MAX_ALLOCATION_SIZE=19_511_627_776

[DEBUG] ggml_extend.hpp:1550 - unet compute buffer size: 11254.99 MB(VRAM)
  |==================================================| 20/20 - 18.72s/it
[INFO ] stable-diffusion.cpp:2158 - sampling completed, taking 379.44s
[INFO ] stable-diffusion.cpp:2166 - generating 1 latent images completed, taking 379.50s
[INFO ] stable-diffusion.cpp:2169 - decoding 1 latents
[DEBUG] ggml_extend.hpp:1550 - vae compute buffer size: 17153.00 MB(VRAM)
D:/stable-diffusion.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:796: GGML_ASSERT(ne <= (int64_t)std::numeric_limits<uint32_t>::max()) failed

with tiling, it did not produce a meaningful image. It could be a computational error or the model couldn't handle this resolution.

[DEBUG] ggml_extend.hpp:1550 - unet compute buffer size: 11254.99 MB(VRAM)
  |==================================================| 20/20 - 18.74s/it
[INFO ] stable-diffusion.cpp:2158 - sampling completed, taking 379.92s
[INFO ] stable-diffusion.cpp:2166 - generating 1 latent images completed, taking 379.98s
[INFO ] stable-diffusion.cpp:2169 - decoding 1 latents
[DEBUG] stable-diffusion.cpp:1521 - VAE Tile size: 32x32
[DEBUG] ggml_extend.hpp:817  - num tiles : 15, 15
[DEBUG] ggml_extend.hpp:818  - optimal overlap : 0.500000, 0.500000 (targeting 0.500000)
[DEBUG] ggml_extend.hpp:851  - tile work buffer size: 0.77 MB
[INFO ] ggml_extend.hpp:864  - processing 225 tiles
[DEBUG] ggml_extend.hpp:1550 - vae compute buffer size: 416.02 MB(VRAM)
  |==================================================| 225/225 - 2.92it/s
[DEBUG] stable-diffusion.cpp:1547 - computing vae decode graph completed, taking 77.58s
[INFO ] stable-diffusion.cpp:2179 - latent 1 decoded, taking 77.59s
[INFO ] stable-diffusion.cpp:2183 - decode_first_stage completed, taking 77.59s
[INFO ] stable-diffusion.cpp:2475 - generate_image completed in 457.98s
save result PNG image to 'output.png'

then I tried 1792x1792 with conv-direct, and it produced yet another error:

[DEBUG] ggml_extend.hpp:1550 - unet compute buffer size: 7005.58 MB(VRAM)
  |==================================================| 20/20 - 12.63s/it
[INFO ] stable-diffusion.cpp:2158 - sampling completed, taking 256.20s
[INFO ] stable-diffusion.cpp:2166 - generating 1 latent images completed, taking 256.26s
[INFO ] stable-diffusion.cpp:2169 - decoding 1 latents
[DEBUG] ggml_extend.hpp:1550 - vae compute buffer size: 10584.77 MB(VRAM)
D:/stable-diffusion.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:6132: GGML_ASSERT(d_D->size >= d_buf_offset + d_sz * ne02 * ne03) failed

which is wierd, because previously a larger unet buffer was allocated successfully(11254.99 MB > 10584.77 MB). I ran the same parameters three times and they produced the same error.
then I tried 1664x1664 with conv-direct, didn't produce a meaningful image, but here's another twist:

// without vae-conv-direct
[DEBUG] ggml_extend.hpp:1550 - unet compute buffer size: 5403.69 MB(VRAM)
  |==================================================| 20/20 - 10.28s/it
[INFO ] stable-diffusion.cpp:2158 - sampling completed, taking 208.31s
[INFO ] stable-diffusion.cpp:2166 - generating 1 latent images completed, taking 208.37s
[INFO ] stable-diffusion.cpp:2169 - decoding 1 latents
[DEBUG] ggml_extend.hpp:1550 - vae compute buffer size: 26406.91 MB(VRAM)
[DEBUG] stable-diffusion.cpp:1547 - computing vae decode graph completed, taking 26.51s
[INFO ] stable-diffusion.cpp:2179 - latent 1 decoded, taking 26.51s
[INFO ] stable-diffusion.cpp:2183 - decode_first_stage completed, taking 26.51s
[INFO ] stable-diffusion.cpp:2475 - generate_image completed in 235.29s
save result PNG image to 'output.png'

the allocation succeed(on GPU-Z and task manager) for some reason, but didn't produce a meaningful image:

and similar result at 1536x1536 without vae-conv-direct.

bonus:

// 2048x2048, no vae-conv-direct
[DEBUG] ggml_extend.hpp:1550 - unet compute buffer size: 11254.99 MB(VRAM)
  |==================================================| 20/20 - 18.72s/it
[INFO ] stable-diffusion.cpp:2158 - sampling completed, taking 379.27s
[INFO ] stable-diffusion.cpp:2166 - generating 1 latent images completed, taking 379.33s
[INFO ] stable-diffusion.cpp:2169 - decoding 1 latents
ggml_vulkan: Device memory allocation of size 19327352832 failed.
ggml_vulkan: vk::Device::allocateMemory: ErrorOutOfDeviceMemory
[ERROR] ggml_extend.hpp:71   - ggml_gallocr_reserve_n: failed to allocate Vulkan0 buffer of size 37386452996
[ERROR] ggml_extend.hpp:1543 - vae: failed to allocate the compute buffer
Segmentation fault

[jeffbolznv]

Here's the log from master-35843c7

For some of these very large convolutions to work, you probably need the very latest ggml (synced a couple days ago).

It's interesting that you can allocate larger than the max memory allocation size, but maybe more interesting/surprising that you can go beyond the reported maxBufferSize. I'll add an env var to override max buffer size in ggml-org/llama.cpp#16354.