ggml-backend-meta: multi buffers are unsupported leading to vulkan segfault

[mattngaw]

Name and Version

version: 8863 (9789512)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux

GGML backends

Vulkan

Hardware

Intel(R) Xeon(R) CPU E5-2630L v3
2× AMD Radeon RX 480 (Polaris10, 4 GB each, 8 GB aggregate VRAM)
Linux 6.8.0-110-generic (Ubuntu 24.04)
Mesa 25.2.8 / RADV driver
No peer-to-peer; PCIe 3.0 x16

Models

Llama 3.2 3B (unsloth/Llama-3.2-3B-Instruct-GGUF:Q4_K_M)

Problem description & steps to reproduce

Summary

llama-cli segfaults during model load when using --split-mode tensor on Vulkan unless the user explicitly passes -c with a "small" context size.

Repro

./build/bin/llama-cli \
  -hf unsloth/Llama-3.2-3B-Instruct-GGUF:Q4_K_M \
  -ngl 99 -dev Vulkan0,Vulkan1 -sm tensor -fa 1

Result: Segmentation fault (core dumped) after "Loading model..." spinner.

Workaround

Any small -c avoids the crash, this works fine:

./build/bin/llama-cli ... -c 2048

First Bad Commit

No response

Relevant log output

Logs

$ gdb --args ./build/bin/llama-cli -hf unsloth/Llama-3.2-3B-Instruct-GGUF:Q4_K_M     -ngl 99 -dev Vulkan0,Vulkan1 -sm tensor -fa 1
GNU gdb (Ubuntu 15.1-1ubuntu1~24.04.1) 15.1
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build/bin/llama-cli...
(No debugging symbols found in ./build/bin/llama-cli)
(gdb) r
Starting program: /home/matto/llama.cpp/build/bin/llama-cli -hf unsloth/Llama-3.2-3B-Instruct-GGUF:Q4_K_M -ngl 99 -dev Vulkan0,Vulkan1 -sm tensor -fa 1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe29ff6c0 (LWP 366687)]
[New Thread 0x7fffe21fe6c0 (LWP 366688)]
[New Thread 0x7fffe18bc6c0 (LWP 366689)]
[New Thread 0x7fffe0f7a6c0 (LWP 366690)]
[New Thread 0x7fffd3fff6c0 (LWP 366692)]
[Thread 0x7fffd3fff6c0 (LWP 366692) exited]

[New Thread 0x7fffd3fff6c0 (LWP 366693)]
[Thread 0x7fffd3fff6c0 (LWP 366693) exited]
[New Thread 0x7fffd3fff6c0 (LWP 366694)]
[Thread 0x7fffd3fff6c0 (LWP 366694) exited]
[New Thread 0x7fffd3fff6c0 (LWP 366695)]
Loading model... |[New Thread 0x7fffd37fe6c0 (LWP 366696)]
/[New Thread 0x7fffd08886c0 (LWP 366743)]
[Thread 0x7fffd08886c0 (LWP 366743) exited]
[New Thread 0x7fffd08886c0 (LWP 366744)]
[Thread 0x7fffd08886c0 (LWP 366744) exited]
[New Thread 0x7fffd08886c0 (LWP 366745)]
[Thread 0x7fffd08886c0 (LWP 366745) exited]
[New Thread 0x7fffd08886c0 (LWP 366746)]
[Thread 0x7fffd08886c0 (LWP 366746) exited]
-
Thread 1 "llama-cli" received signal SIGSEGV, Segmentation fault.
0x00007ffff3970fb0 in ggml_backend_vk_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/matto/llama.cpp/build/bin/libggml-vulkan.so.0
(gdb) bt full
#0  0x00007ffff3970fb0 in ggml_backend_vk_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/matto/llama.cpp/build/bin/libggml-vulkan.so.0
No symbol table info available.
#1  0x00007ffff797fcb5 in ggml_backend_meta_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/matto/llama.cpp/build/bin/libggml-base.so.0
No symbol table info available.
#2  0x00007ffff7975877 in ggml_backend_sched_graph_compute_async () from /home/matto/llama.cpp/build/bin/libggml-base.so.0
No symbol table info available.
#3  0x00007ffff76bc461 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/matto/llama.cpp/build/bin/libllama.so.0
No symbol table info available.
#4  0x00007ffff76be574 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/matto/llama.cpp/build/bin/libllama.so.0
No symbol table info available.
#5  0x00007ffff76c5920 in llama_context::decode(llama_batch const&) () from /home/matto/llama.cpp/build/bin/libllama.so.0
No symbol table info available.
#6  0x00007ffff76c73ef in llama_decode () from /home/matto/llama.cpp/build/bin/libllama.so.0
No symbol table info available.
#7  0x00007ffff7bfa59f in common_init_from_params(common_params&) () from /home/matto/llama.cpp/build/bin/libllama-common.so.0
No symbol table info available.
#8  0x0000555555673068 in server_context_impl::load_model(common_params&) ()
No symbol table info available.
#9  0x00005555555a082e in main ()
No symbol table info available.

[mattngaw]

gdb backtrace on the debug llama-cli:

Logs

$ gdb --args ./build-dbg/bin/llama-cli -hf unsloth/Llama-3.2-3B-Instruct-GGUF:Q4_K_M -ngl 99 -dev Vulkan0,Vulkan1 -sm tensor -fa 1
GNU gdb (Ubuntu 15.1-1ubuntu1~24.04.1) 15.1
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./build-dbg/bin/llama-cli...
(gdb) r
Starting program: /home/matto/llama.cpp/build-dbg/bin/llama-cli -hf unsloth/Llama-3.2-3B-Instruct-GGUF:Q4_K_M -ngl 99 -dev Vulkan0,Vulkan1 -sm tensor -fa 1
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe2bff6c0 (LWP 366916)]
[New Thread 0x7fffe23fe6c0 (LWP 366917)]
[New Thread 0x7fffe1abc6c0 (LWP 366918)]
[New Thread 0x7fffe117a6c0 (LWP 366919)]
[New Thread 0x7fffe09796c0 (LWP 366920)]
[Thread 0x7fffe09796c0 (LWP 366920) exited]

[New Thread 0x7fffe09796c0 (LWP 366921)]
[Thread 0x7fffe09796c0 (LWP 366921) exited]
[New Thread 0x7fffe09796c0 (LWP 366922)]
[Thread 0x7fffe09796c0 (LWP 366922) exited]
[New Thread 0x7fffe09796c0 (LWP 366923)]
Loading model... |[New Thread 0x7fffcbbff6c0 (LWP 366924)]
-[New Thread 0x7fffcb3fe6c0 (LWP 366958)]
[Thread 0x7fffcb3fe6c0 (LWP 366958) exited]
[New Thread 0x7fffcb3fe6c0 (LWP 366959)]
[Thread 0x7fffcb3fe6c0 (LWP 366959) exited]
[New Thread 0x7fffcb3fe6c0 (LWP 366960)]
[Thread 0x7fffcb3fe6c0 (LWP 366960) exited]
[New Thread 0x7fffcb3fe6c0 (LWP 366961)]
[Thread 0x7fffcb3fe6c0 (LWP 366961) exited]
-
Thread 1 "llama-cli" received signal SIGSEGV, Segmentation fault.
__gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0x39) at /usr/include/c++/13/ext/atomicity.h:111
111           __atomic_add(__mem, __val);
(gdb) bt full
#0  __gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0x39) at /usr/include/c++/13/ext/atomicity.h:111
No locals.
#1  std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_add_ref_copy (this=0x31) at /usr/include/c++/13/bits/shared_ptr_base.h:152
No locals.
#2  std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count (__r=..., this=<synthetic pointer>) at /usr/include/c++/13/bits/shared_ptr_base.h:1078
No locals.
#3  std::__shared_ptr<vk_buffer_struct, (__gnu_cxx::_Lock_policy)2>::__shared_ptr (this=<synthetic pointer>) at /usr/include/c++/13/bits/shared_ptr_base.h:1522
No locals.
#4  std::shared_ptr<vk_buffer_struct>::shared_ptr (this=<synthetic pointer>) at /usr/include/c++/13/bits/shared_ptr.h:204
No locals.
#5  ggml_vk_tensors_overlap (elementwise=false, b=<optimized out>, a=0x7ffe1f485480) at /home/matto/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:14319
        a_buf_ctx = 0x555558eca260
        a_buf = <optimized out>
        b_buf_ctx = 0x555555747e50
        b_buf = <optimized out>
        a_buf_ctx = <optimized out>
        a_buf = <optimized out>
        b_buf_ctx = <optimized out>
        b_buf = <optimized out>
        a_base = <optimized out>
        a_size = <optimized out>
        b_base = <optimized out>
        b_size = <optimized out>
#6  ggml_backend_vk_graph_compute (backend=<optimized out>, cgraph=0x555559bbcc20) at /home/matto/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:14690
        src = 0x7ffe1f485480
        s = 0
        k = 0
        dst = 0x7fff899dd160
        j = 0
        bits = <optimized out>
        output_nodes = {0x7fff899dd160, 0x0}
        need_disable = <optimized out>
        is_topk_moe_single_row = <optimized out>
        almost_ready = <optimized out>
        submit = <optimized out>
        enqueued = <optimized out>
        op_srcs_fused_elementwise = {false, false, false, false, 37, false, false, false, 42, false, false, false}
        fusion_string = <optimized out>
        i = <optimized out>
        ctx = 0x555555808c30
        last_node = <optimized out>
        first_node_in_batch = <optimized out>
        submit_node_idx = 0
        compute_ctx = std::shared_ptr<vk_context_struct> (use count 2, weak count 6) = {get() = 0x5555598dc5d0}
        nodes_per_submit = 100
        submitted_nodes = <optimized out>
        submit_count = <optimized out>
        mul_mat_bytes = 4829184
        total_mul_mat_bytes = 4829184
        mul_mat_bytes_per_submit = <optimized out>
#7  0x00007ffff7e90604 in ggml_backend_meta_graph_compute (backend=<optimized out>, cgraph=<optimized out>) at /usr/include/c++/13/bits/stl_vector.h:1126
        bcj = <optimized out>
        status = <optimized out>
        j = 0
        i = 0
        n_backends = <optimized out>
        backend_ctx = 0x555555af4fa0
        needs_rebuild = <optimized out>
        max_nnodes_raised = <optimized out>
        iga = 0
        ina = 0
        get_node_aux = {__backend_ctx = @0x7fffffff2788, __ina = @0x7fffffff2798}
        set_tmp_data = {__backend_ctx = @0x7fffffff2788}
        get_cgraph_aux = {__backend_ctx = @0x7fffffff2788, __iga = @0x7fffffff2790}
        allreduce_fallback = {__n_backends =
./gdb/dwarf2/expr.c:597: internal-error: coerce_pieced_ref: Assertion `closure->pieces.size () == 1' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.

I intend to continue digging into this myself, if anyone could verify reproducibility that'd be great.

[LostRuins]

@mattngaw I can reproduce on vulkan with Qwen3 4B (full offload) and it segfaulted, somewhere in ggml_vk_tensors_overlap (elementwise=false, b=0x22c1e16b3a0, a=0x22cac9a9ba0) at

llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp

Line 14495 in 63d93d1

vk_buffer b_buf = b_buf_ctx->dev_buffer;

Testing with a single GPU, laptop rtx 4090.

#19378 (comment)

[TheBlueMatt]

This may be fixed by TheBlueMatt@6e113cb someone needs to clean it up and upstream at some point...

[d-shehu]

Any chance this will eventually work with AMD GPUs using Vulkan or is explicitly Nvidia-centric?

Would it also work with RPC using mix GPUs (RADv Vulkan + Nvidia Cuda)?

[d-shehu]

I just tested b9494 and this still crashes with OOM, alloc_tensor_range, with every model (example: Qwen 3.6 27b and 3.5 122b).

"Layer" or "row" work fine.

[netrunnereve]

I'm seeing this as well and finally got some time to look into it. Basically what happens is that when the KV cache gets too big the Vulkan backend allocates multiple buffers to fit it in memory rather than a single buffer, for me that triggers around 1GB or so. If you set GGML_VK_MEMORY_LOGGER=1 and mess with the context size then there's a point where the backend will decide to split up the buffers.

I think CUDA is able to allocate large buffers in one go so it doesn't have this problem, but I can't test this. Anyways when the backend uses multiple buffers to store all the data a ggml multi_buffer gets created, which wraps multiple buffers together in a single object (the tensors are still assigned to actual buffers). I think it's done that way so that ggml actually knows that there are several buffers and that they need to be deleted together and all that. The multi_buffer itself doesn't have a valid context and we get our segfault.

#6  0x00007ffff3e7d4ba in ggml_vk_tensors_overlap (a=0x7fffae31b480, b=0x555555d812a0, elementwise=false)
    at <eve's dir>/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:15620
        a_buf_ctx = 0x5555556ef7e0
        a_buf = std::shared_ptr<vk_buffer_struct> (use count 2, weak count 0) = {get() = 0x555555ba92c0}
        b_buf_ctx = 0x5555556edbc0
        b_buf = <error reading variable: Cannot access memory at address 0x59> <--- SEGFAULT

Here b is a multi buffer. @TheBlueMatt's patch seems to work as it uses the original Vulkan source of the buffer instead of the multi buffer. However it depends on the fact that the tensor has a valid view src (is that always the case?) that links to the original buffer.

If we have a small KV cache and don't have a multi buffer then the meta backend delivers a regular Vulkan buffer (I checked). I'm still looking into it but basically the meta backend was created assuming that there's only one buffer per ggml context.

(gdb) print *(ggml_backend_vk_buffer_context *)b->buffer->context
$68 = {device = <error reading variable: Cannot access memory at address 0xa>, 
  dev_buffer = <error reading variable: Cannot access memory at address 0x59>, 
  name = <error: Cannot access memory at address 0x7fff00000001>}
(gdb) print *(ggml_backend_vk_buffer_context *)b->view_src->buffer->context
$69 = {device = std::weak_ptr<vk_device_struct> (use count 13, weak count 6) = {get() = 0x5555556f0b40}, 
  dev_buffer = std::shared_ptr<vk_buffer_struct> (use count 1, weak count 0) = {get() = 0x5555556ee020}, 
  name = "Vulkan0"}

So this explains why it crashes but don't know the code well enough to actually confirm that his patch fixes this properly.

[JohnCMcDonough]

I think I’m hitting the same bug here, once I push -c / --ctx-size high enough. I'm still figuring out llama.cpp, so I hope this is helpful.

Behavior so far:

• --ctx-size 32768: survives
• --ctx-size 64000: segfaults

Hardware:

• OS: Bluefin 44 / Fedora 44 base
• Kernel: 6.19.14-101.fc44.x86_64
• CPU: AMD Ryzen 7 9800X3D
• RAM: 60.4 GiB
• GPUs:
  • AMD Radeon AI PRO R9700 (gfx1201), 32 GB VRAM
  • AMD Radeon AI PRO R9700 (gfx1201), 32 GB VRAM
• Vulkan driver: Mesa RADV
• llama-server --version: 9660 (7dad2f1a1)

I can reproduce it reliably with:

llama-server \
  -hf unsloth/Qwen3.6-27B-MTP-GGUF:Q4_K_M \
  --host 127.0.0.1 --port 9090 \
  --device Vulkan1,Vulkan2 \
  -sm tensor -fa on \
  --ctx-size 64000 \
  --fit off --no-warmup \
  --batch-size 4096 --ubatch-size 1024 \
  -np 1 \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  -bs -cb

(I'm pretty sure backend sampling doesn't work with tensor split though, so that can safely be removed).

With this setup, the process exits with 139. The kernel log shows a segfault in libggml-vulkan.so after model load, around slot initialization.

Full gdb trace

warning: File "/var/home/linuxbrew/.linuxbrew/Cellar/gcc/15.2.0_1/lib/gcc/current/libstdc++.so.6.0.34-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
	add-auto-load-safe-path /var/home/linuxbrew/.linuxbrew/Cellar/gcc/15.2.0_1/lib/gcc/current/libstdc++.so.6.0.34-gdb.py
line to your configuration file "/var/home/john/.config/gdb/gdbinit".
To completely disable this security protection add
	set auto-load safe-path /
line to your configuration file "/var/home/john/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
	info "(gdb)Auto-loading safe path"
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff5fff6c0 (LWP 1322561)]
[Thread 0x7ffff5fff6c0 (LWP 1322561) exited]
[New Thread 0x7ffff5fff6c0 (LWP 1322562)]
[New Thread 0x7fff575ff6c0 (LWP 1322563)]
[New Thread 0x7fff569ff6c0 (LWP 1322564)]
WARNING: radv is not a conformant Vulkan implementation, testing use only.
[New Thread 0x7fff55dff6c0 (LWP 1322565)]
[New Thread 0x7fff551ff6c0 (LWP 1322566)]
WARNING: radv is not a conformant Vulkan implementation, testing use only.
[New Thread 0x7fff47fff6c0 (LWP 1322567)]
[New Thread 0x7fff477fe6c0 (LWP 1322568)]
[New Thread 0x7ffff52426c0 (LWP 1322569)]
[New Thread 0x7fff693ff6c0 (LWP 1322570)]
[Thread 0x7ffff52426c0 (LWP 1322569) exited]
[Thread 0x7fff693ff6c0 (LWP 1322570) exited]
0.00.495.763 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.495.765 I device_info:
0.00.495.771 I   - BLAS    : OpenBLAS (0 MiB, 0 MiB free)
0.00.495.913 I   - Vulkan0 : AMD Ryzen 7 9800X3D 8-Core Processor (RADV RAPHAEL_MENDOCINO) (32972 MiB, 32536 MiB free)
0.00.495.977 I   - Vulkan1 : AMD Radeon AI PRO R9700 (RADV GFX1201) (32624 MiB, 32016 MiB free)
0.00.496.041 I   - Vulkan2 : AMD Radeon AI PRO R9700 (RADV GFX1201) (32624 MiB, 32566 MiB free)
0.00.496.046 I   - CPU     : AMD Ryzen 7 9800X3D 8-Core Processor (61849 MiB, 61849 MiB free)
0.00.496.081 I system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | OPENMP = 1 | REPACK = 1 |
0.00.496.103 I srv          init: running without SSL
0.00.496.130 I srv          init: using 15 threads for HTTP server
0.00.496.274 I srv         start: binding port with default address family
[New Thread 0x7fff693ff6c0 (LWP 1322571)]
0.00.496.402 I srv  llama_server: loading model
0.00.496.410 I srv    load_model: loading model '/var/home/john/.cache/huggingface/hub/models--unsloth--Qwen3.6-27B-MTP-GGUF/snapshots/5cb35eb3dcbf52dbce5f87dbc64df6aaffadcace/Qwen3.6-27B-Q4_K_M.gguf'
[New Thread 0x7ffff52426c0 (LWP 1322572)]
[New Thread 0x7fff67ec26c0 (LWP 1322573)]
[New Thread 0x7fff676c16c0 (LWP 1322574)]
[New Thread 0x7fff6672b6c0 (LWP 1322575)]
[New Thread 0x7fff65f2a6c0 (LWP 1322576)]
[New Thread 0x7fff657296c0 (LWP 1322577)]
[New Thread 0x7fff5b16e6c0 (LWP 1322578)]
[New Thread 0x7fff5a96d6c0 (LWP 1322579)]
[New Thread 0x7fff5a16c6c0 (LWP 1322580)]
[New Thread 0x7fff5996b6c0 (LWP 1322581)]
[New Thread 0x7fff46ffd6c0 (LWP 1322582)]
[New Thread 0x7fff467fc6c0 (LWP 1322583)]
[New Thread 0x7fff45ffb6c0 (LWP 1322584)]
[New Thread 0x7fff457fa6c0 (LWP 1322585)]
[New Thread 0x7fff44ff96c0 (LWP 1322586)]
0.03.666.069 W set_sampler: backend sampling not supported with SPLIT_MODE_TENSOR; using CPU
0.03.666.084 W llama_context: n_ctx_seq (64000) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.03.729.665 I srv    load_model: creating MTP draft context against the target model '/var/home/john/.cache/huggingface/hub/models--unsloth--Qwen3.6-27B-MTP-GGUF/snapshots/5cb35eb3dcbf52dbce5f87dbc64df6aaffadcace/Qwen3.6-27B-Q4_K_M.gguf'
0.03.729.679 W llama_context: n_ctx_seq (64000) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
0.03.738.369 W load_hparams: Qwen-VL models require at minimum 1024 image tokens to function correctly on grounding tasks
0.03.738.370 W load_hparams: if you encounter problems with accuracy, try adding --image-min-tokens 1024
0.03.738.370 W load_hparams: more info: https://github.com/ggml-org/llama.cpp/issues/16842

0.03.934.825 I srv    load_model: loaded multimodal model, '/var/home/john/.cache/huggingface/hub/models--unsloth--Qwen3.6-27B-MTP-GGUF/snapshots/5cb35eb3dcbf52dbce5f87dbc64df6aaffadcace/mmproj-BF16.gguf'
0.03.934.836 I srv    load_model: initializing slots, n_slots = 1

Thread 1 "llama-server" received signal SIGSEGV, Segmentation fault.
0x00007fff695d0d13 in ggml_vk_build_graph(ggml_backend_vk_context*, ggml_cgraph*, int, ggml_tensor*, int, bool, bool, bool)::{lambda(ggml_tensor const*, std::vector<ggml_tensor const*, std::allocator<ggml_tensor const*> > const&)#1}::operator()(ggml_tensor const*, std::vector<ggml_tensor const*, std::allocator<ggml_tensor const*> > const&) const [clone .isra.0] () from /home/linuxbrew/.linuxbrew/Cellar/ggml/0.15.1/libexec/libggml-vulkan.so
#0  0x00007fff695d0d13 in ggml_vk_build_graph(ggml_backend_vk_context*, ggml_cgraph*, int, ggml_tensor*, int, bool, bool, bool)::{lambda(ggml_tensor const*, std::vector<ggml_tensor const*, std::allocator<ggml_tensor const*> > const&)#1}::operator()(ggml_tensor const*, std::vector<ggml_tensor const*, std::allocator<ggml_tensor const*> > const&) const [clone .isra.0] () from /home/linuxbrew/.linuxbrew/Cellar/ggml/0.15.1/libexec/libggml-vulkan.so
#1  0x00007fff695c747f in ggml_vk_build_graph(ggml_backend_vk_context*, ggml_cgraph*, int, ggml_tensor*, int, bool, bool, bool) [clone .isra.0] () from /home/linuxbrew/.linuxbrew/Cellar/ggml/0.15.1/libexec/libggml-vulkan.so
#2  0x00007fff6959c111 in ggml_backend_vk_graph_compute(ggml_backend*, ggml_cgraph*) [clone .lto_priv.0] () from /home/linuxbrew/.linuxbrew/Cellar/ggml/0.15.1/libexec/libggml-vulkan.so
#3  0x00007ffff779f5d3 in ggml_backend_meta_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/linuxbrew/.linuxbrew/opt/ggml/lib/libggml-base.so.0
#4  0x00007ffff77950b5 in ggml_backend_sched_graph_compute_async () from /home/linuxbrew/.linuxbrew/opt/ggml/lib/libggml-base.so.0
#5  0x00007ffff68de8c1 in llama_context::graph_compute(ggml_cgraph*, bool) () from /var/home/linuxbrew/.linuxbrew/Cellar/llama.cpp/9660/bin/../lib/../lib/libllama.so.0
#6  0x00007ffff68e154a in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /var/home/linuxbrew/.linuxbrew/Cellar/llama.cpp/9660/bin/../lib/../lib/libllama.so.0
#7  0x00007ffff68e6e2a in llama_context::decode(llama_batch const&) () from /var/home/linuxbrew/.linuxbrew/Cellar/llama.cpp/9660/bin/../lib/../lib/libllama.so.0
#8  0x00007ffff68e862f in llama_decode () from /var/home/linuxbrew/.linuxbrew/Cellar/llama.cpp/9660/bin/../lib/../lib/libllama.so.0
#9  0x00007ffff6e01f11 in common_context_can_seq_rm(llama_context*) () from /var/home/linuxbrew/.linuxbrew/Cellar/llama.cpp/9660/bin/../lib/../lib/libllama-common.so.0
#10 0x00007ffff798436c in server_context_impl::load_model(common_params&) () from /var/home/linuxbrew/.linuxbrew/Cellar/llama.cpp/9660/bin/../lib/libllama-server-impl.so
#11 0x00007ffff78def70 in llama_server(int, char**) () from /var/home/linuxbrew/.linuxbrew/Cellar/llama.cpp/9660/bin/../lib/libllama-server-impl.so
#12 0x00007ffff7208681 in __libc_start_call_main () from /lib64/libc.so.6
#13 0x00007ffff7208798 in __libc_start_main_impl () from /lib64/libc.so.6
#14 0x00005555555562c5 in _start ()

[netrunnereve]

So this explains why it crashes but don't know the code well enough to actually confirm that his patch fixes this properly.

Okay this patch is only going to work if the tensor in the multi buffer is a view tensor. If it's the original source of the data then it's not going to have a view_src and the patched function isn't going to return the correct Vulkan buffer.

I dug into this and how the multi buffer works is that it basically allocates multiple buffers on the backend when you do a ggml_backend_alloc, and it returns a multi buffer object instead of a backend buffer. The multi buffer in turn contains an array of pointers to the backend buffers and it has some functions allowing you to say free or clear all the associated buffers in one go. In fact multi buffers were actually created specifically by @0cc4m for the Vulkan backend a long time ago.

Since with multi buffers it's possible for tensors to be stored in different physical buffers, during the allocation process each tensor has its associated tensor->buffer set to identify where it's actually stored. The tensors don't know or care if it's in a multi buffer and when we run the model the backend just goes to tensor->buffer to get the actual location of the data.

Now the meta backend used for tensor parallel is a virtual backend that picks up the graph like any other backend and internally splits it up so that it runs in parallel across different backend devices. It does a ggml_backend_alloc for each backend and it keeps track of the returned buffer. That buffer is then passed on to the meta backend context tensors under the assumption that each backend has a single buffer for each context. So in our case with a multi buffer tensor->buffer ends up pointing to a multi buffer and we get our segfault as the Vulkan backend expects a single Vulkan buffer. I thought of modifying the Vulkan backend to make it support multi buffers but that requires it to know which buffer in the multi buffer is storing that specific tensor, and we don't know that.

What I think we can do is:

• Have some sort of way to store the context tensor buffers during ggml_backend_alloc when it returns a multi buffer, and later when we init the meta backend transfer those buffers over to the new meta tensors. Obviously we would have to ensure that the meta tensor being initialized is the same as the one that was allocated on the backend in the first place.
• Or have the multi buffer object not only point to the buffers it contains but also store the list of tensors that are held inside each buffer.

Of course I might be missing something here and I'm not going to pretend that I understand the meta backend fully as I don't!