Bug description
When running llama-server from BeeLlama with KVarN KV cache and CUDA, using ornith-1.0-35b-1M-MTP-APEX-Compact.gguf at --ctx-size 1048576 on an RTX 3090, I get a reproducible crash in the ggml_cuda_fattn_kvarn_decode_launch kernel once the actual context length approaches ~900k tokens.
Important details:
--ctx-size is set to 1048576.- With the same model and runtime options:
- Inference is stable for prompts / conversations up to around 500k tokens of total context.
- When I continue the same workload and the total context reaches about 900k tokens, the server crashes.
- If I reduce
--ctx-size to 524288 with the same configuration, I do not see this crash (but of course the model cannot go beyond ~524k tokens in that case).
So the issue appears to be related to the interaction between very long effective context (~900k tokens) + KVarN + CUDA flash attention at ctx-size 1048576, rather than a generic environment or model problem.
Stack trace
GDB backtrace at the point of failure (trimmed to the relevant part):
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007615f1d10813 in __GI___wait4 (pid=1546222, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
#0 0x00007615f1d10813 in __GI___wait4 (pid=1546222, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
#1 0x00007615f1f43983 in ggml_print_backtrace () from /home/user/beellama-j4/build/bin/libggml-base.so.0
#2 0x00007615f1f43b2b in ggml_abort () from /home/user/beellama-j4/build/bin/libggml-base.so.0
#3 0x00007615e6d4a527 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from /home/user/beellama-j4/build/bin/libggml-cuda.so.0
#4 0x00007615e757a7ab in void ggml_cuda_fattn_kvarn_decode_launch<256, 3, 3>(ggml_cuda_fattn_kvarn_decode_args const&) () from /home/user/beellama-j4/build/bin/libggml-cuda.so.0
#5 0x00007615e6cc52bd in bool ggml_cuda_flash_attn_ext_kvarn_decode_d<256>(ggml_backend_cuda_context&, ggml_tensor*, ggml_cuda_fattn_kvarn_plan const&) () from /home/user/beellama-j4/build/bin/libggml-cuda.so.0
#6 0x00007615e6cc9bd2 in ggml_cuda_flash_attn_ext_kvarn(ggml_backend_cuda_context&, ggml_tensor*, ggml_cuda_fattn_kvarn_entry_path) () from /home/user/beellama-j4/build/bin/libggml-cuda.so.0
#7 0x00007615e6cdf5bc in ggml_cuda_flash_attn_ext_tail(ggml_backend_cuda_context&, ggml_tensor*) () from /home/user/beellama-j4/build/bin/libggml-cuda.so.0
#8 00007615e6d60989 in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/user/beellama-j4/build/bin/libggml-cuda.so.0
#9 0x00007615f1f63972 in ggml_backend_sched_graph_compute_async () from /home/user/beellama-j4/build/bin/libggml-base.so.0
#10 0x00007615f130f7b1 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/user/beellama-j4/build/bin/libllama.so.0
#11 0x00007615f13135d4 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) () from /home/user/beellama-j4/build/bin/libllama.so.0
#12 0x00007615f131bb8b in llama_context::decode(llama_batch const&) () from /home/user/beellama-j4/build/bin/libllama.so.0
#13 0x00007615f131d4cf in llama_decode () from /home/user/beellama-j4/build/bin/libllama.so.0
#14 0x00007615f25bebbd in server_context_impl::decode(int&, int, llama_batch&) () from /home/user/beellama-j4/build/bin/libllama-server-impl.so
#15 0x00007615f25c0312 in server_context_impl::update_slots() () from /home/user/beellama-j4/build/bin/libllama-server-impl.so
#16 0x00007615f255fcf1 in server_queue::start_loop(long) () from /home/user/beellama-j4/build/bin/libllama-server-impl.so
#17 0x00007615f24fff2a in llama_server(common_params&, int, char**) () from /home/user/beellama-j4/build/bin/libllama-server-impl.so
#18 0x00007615f2500ce6 in llama_server(int, char**) () from /home/user/beellama-j4/build/bin/libllama-server-impl.so
Steps to reproduce
- Build BeeLlama with CUDA support:
git clone https://github.com/Anbeeld/beellama.cpp.git
cd beellama.cpp
mkdir build && cd build
cmake .. -DGGML_CUDA=ON -DGGML_CUDA_F16=ON
make -j$(nproc)
- Launch
llama-server with ctx-size set to 1048576 (this is the configuration that eventually crashes):
./bin/llama-server \
--model /path/to/models/ornith-1.0-35b-1M-MTP-APEX-Compact.gguf \
--ctx-size 1048576 \
--ngl <NGL_VALUE> \
--threads <N_THREADS> \
--cache-type-k <CACHE_TYPE_K> \
--cache-type-v <CACHE_TYPE_V> \
--flash-attn \
<ANY_OTHER_RELEVANT_FLAGS>
-
Run a workload that gradually builds up a long context (e.g. long prompt or long multi-turn conversation).
-
Observe that:
- Up to around 500k tokens of total context, the model runs stably.
- When the total context reaches around 900k tokens (still below the
1048576 context limit advertised for this model), llama-server crashes, with the backtrace shown above.
- As a comparison, restart
llama-server with a smaller context size:
./bin/llama-server \
--model /path/to/models/ornith-1.0-35b-1M-MTP-APEX-Compact.gguf \
--ctx-size 524288 \
--ngl <NGL_VALUE> \
--threads <N_THREADS> \
--cache-type-k <CACHE_TYPE_K> \
--cache-type-v <CACHE_TYPE_V> \
--flash-attn \
<ANY_OTHER_RELEVANT_FLAGS>
- With
--ctx-size 524288 (and otherwise the same configuration), the crash does not occur under the same kind of workload (the model cannot exceed ~524k tokens of context, but there is no CUDA kernel failure).
Expected behavior
llama-server should not crash when the total context is still below the configured --ctx-size 1048576 and below the 1M context that ornith-1.0-35b-1M-MTP-APEX-Compact.gguf is designed to support. If there is an internal hard limit (e.g. around 524k or 900k tokens) coming from the KVarN + flash-attention CUDA path, it would be helpful to either handle that limit gracefully, or document the effective maximum safe context for this combination of features.
Actual behavior
- With
--ctx-size 1048576: inference is stable up to ~500k tokens of total context, then crashes around ~900k tokens during decoding with a ggml_cuda_error and the backtrace above.
- With
--ctx-size 524288: the same configuration (model, K/V cache types, flash-attention) does not crash under similar usage patterns; the effective context is capped around 524k tokens.
Environment
- BeeLlama version / commit:
- Build type:
cmake .. -DGGML_CUDA=ON -DGGML_CUDA_F16=ON
- OS:
- Kernel:
- GPU: NVIDIA GeForce RTX 3090 24 GB
- NVIDIA driver version: <DRIVER_VERSION>
- CUDA toolkit version: <CUDA_VERSION>
- CPU:
- System RAM: <RAM_GB> GB
- Model: ornith-1.0-35b-1M-MTP-APEX-Compact.gguf
- Llama server options (full command line):
<FULL_LLAMA_SERVER_COMMAND_HERE>
Additional notes
- Disabling KVarN and/or flash attention (e.g. switching
--cache-type-k/v to a standard quant like q8_0, or running without --flash-attn) appears to avoid this crash, which points to the KVarN + flash-attention CUDA path in BeeLlama.
- The fact that smaller
--ctx-size = 524288 is stable, while --ctx-size = 1048576 is stable up to ~500k tokens but crashes around ~900k, suggests a boundary or sizing issue specific to very long contexts in the KVarN flash-attention kernels, rather than a simple VRAM OOM.
- I can test patches or debug builds if needed.
Bug description
When running
llama-serverfrom BeeLlama with KVarN KV cache and CUDA, usingornith-1.0-35b-1M-MTP-APEX-Compact.ggufat--ctx-size 1048576on an RTX 3090, I get a reproducible crash in theggml_cuda_fattn_kvarn_decode_launchkernel once the actual context length approaches ~900k tokens.Important details:
--ctx-sizeis set to1048576.--ctx-sizeto524288with the same configuration, I do not see this crash (but of course the model cannot go beyond ~524k tokens in that case).So the issue appears to be related to the interaction between very long effective context (~900k tokens) + KVarN + CUDA flash attention at
ctx-size 1048576, rather than a generic environment or model problem.Stack trace
GDB backtrace at the point of failure (trimmed to the relevant part):
Steps to reproduce
llama-serverwithctx-sizeset to1048576(this is the configuration that eventually crashes):Run a workload that gradually builds up a long context (e.g. long prompt or long multi-turn conversation).
Observe that:
1048576context limit advertised for this model),llama-servercrashes, with the backtrace shown above.llama-serverwith a smaller context size:--ctx-size 524288(and otherwise the same configuration), the crash does not occur under the same kind of workload (the model cannot exceed ~524k tokens of context, but there is no CUDA kernel failure).Expected behavior
llama-servershould not crash when the total context is still below the configured--ctx-size 1048576and below the 1M context thatornith-1.0-35b-1M-MTP-APEX-Compact.ggufis designed to support. If there is an internal hard limit (e.g. around 524k or 900k tokens) coming from the KVarN + flash-attention CUDA path, it would be helpful to either handle that limit gracefully, or document the effective maximum safe context for this combination of features.Actual behavior
--ctx-size 1048576: inference is stable up to ~500k tokens of total context, then crashes around ~900k tokens during decoding with aggml_cuda_errorand the backtrace above.--ctx-size 524288: the same configuration (model, K/V cache types, flash-attention) does not crash under similar usage patterns; the effective context is capped around 524k tokens.Environment
cmake .. -DGGML_CUDA=ON -DGGML_CUDA_F16=ONAdditional notes
--cache-type-k/vto a standard quant likeq8_0, or running without--flash-attn) appears to avoid this crash, which points to the KVarN + flash-attention CUDA path in BeeLlama.--ctx-size = 524288is stable, while--ctx-size = 1048576is stable up to ~500k tokens but crashes around ~900k, suggests a boundary or sizing issue specific to very long contexts in the KVarN flash-attention kernels, rather than a simple VRAM OOM.