Name and Version
lama.cpp-b8646
Operating systems
Linux
Which llama.cpp modules do you know to be affected?
libllama (core library)
Command line
./llama-batched-bench -m "../models/Llama-2-7b.Q4_0.gguf" -ts 0,17,16 -fa on -c 65000 -npp 256 -ntg 2000 -npl 20,24,26,27,28,30 -kvu -ctk q8_0 -ctv q8_0 -p "who is napoleon?"
Problem description & steps to reproduce
When running llama.cpp with many clients (in batch mode) memory is consumed permanently during token generation. I have seen the same problem with prompt processing, but it usually short, so the problem doesn't manifest itself too loud. But when many requests are served and generation can be relatively long (thousands of tokens) for each request the issue eventually leads to a crash because of memory limit is hit.
To reproduce the memory leak it is enough to run the llama-batched-bench with the command line from above. Start the bench and periodically run "nvidia-smi -q -d MEMORY", you'll see that memory consumption grows approximately megabyte per second (in bigger steps, may be 5mb). The parameters were selected to get a long time load, with it the memory leak is obvious when your bench just crashes, but of course, it depends on the memory you have. Another point of the parameters was to get to a timeout, but with llama-2 it was unsuccessful (see below).
Next problem is the timeout. The same llama-batched-bench (but with different model) eventually leads to a crash with CUDA timeout. The logs clearly show the line number within a CUDA source where "the launch timed out and was terminated". I can not reproduce it with the small model (llama-2), tried to run with it for a longer time than the log shows, but had no success, may be the timeout is due to some processing that is actual for bigger models, hope that the CUDA source line would help.
And another point, when trying to get maximum performance for maximum requests it is required to reserve as much memory as possible, so, splitting it across GPUs helps, but it is a bit of pain when for every model there is a quest of how to properly split it and how large a context could be. Server (and all other llama.cpp programs) just crash because of out of memory and do not tell you any sign of how much of context to remove or how many layers to move, just plain trial and error for many times with every model (because models have specifics, like additional memory requirement per parallel task).
It would be great if llama.cpp just logged the memory size it expects to use with provided parameters. Currently it logs the amount, but split into some parts and only after a successful start, not before, when it is really needed.
First Bad Commit
No response
Relevant log output
Logs
| PP | TG | B | N_KV | T_PP s | S_PP t/s | T_TG s | S_TG t/s | T s | S t/s |
|-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
| 10 | 2000 | 10 | 20010 | 0.081 | 123.41 | 197.348 | 101.34 | 197.429 | 101.35 |
| 10 | 2000 | 11 | 22010 | 0.072 | 139.55 | 204.940 | 107.35 | 205.012 | 107.36 |
| 10 | 2000 | 12 | 24010 | 0.072 | 139.54 | 211.642 | 113.40 | 211.714 | 113.41 |
llama.cpp-b8646/ggml/src/ggml-cuda/ggml-cuda.cu:98: CUDA error
CUDA error: the launch timed out and was terminated
current device: 2, in function ggml_backend_cuda_synchronize at llama.cpp-b8646/ggml/src/ggml-cuda/ggml-cuda.cu:2924
cudaStreamSynchronize(cuda_ctx->stream())
Name and Version
lama.cpp-b8646
Operating systems
Linux
Which llama.cpp modules do you know to be affected?
libllama (core library)
Command line
Problem description & steps to reproduce
When running llama.cpp with many clients (in batch mode) memory is consumed permanently during token generation. I have seen the same problem with prompt processing, but it usually short, so the problem doesn't manifest itself too loud. But when many requests are served and generation can be relatively long (thousands of tokens) for each request the issue eventually leads to a crash because of memory limit is hit.
To reproduce the memory leak it is enough to run the llama-batched-bench with the command line from above. Start the bench and periodically run "nvidia-smi -q -d MEMORY", you'll see that memory consumption grows approximately megabyte per second (in bigger steps, may be 5mb). The parameters were selected to get a long time load, with it the memory leak is obvious when your bench just crashes, but of course, it depends on the memory you have. Another point of the parameters was to get to a timeout, but with llama-2 it was unsuccessful (see below).
Next problem is the timeout. The same llama-batched-bench (but with different model) eventually leads to a crash with CUDA timeout. The logs clearly show the line number within a CUDA source where "the launch timed out and was terminated". I can not reproduce it with the small model (llama-2), tried to run with it for a longer time than the log shows, but had no success, may be the timeout is due to some processing that is actual for bigger models, hope that the CUDA source line would help.
And another point, when trying to get maximum performance for maximum requests it is required to reserve as much memory as possible, so, splitting it across GPUs helps, but it is a bit of pain when for every model there is a quest of how to properly split it and how large a context could be. Server (and all other llama.cpp programs) just crash because of out of memory and do not tell you any sign of how much of context to remove or how many layers to move, just plain trial and error for many times with every model (because models have specifics, like additional memory requirement per parallel task).
It would be great if llama.cpp just logged the memory size it expects to use with provided parameters. Currently it logs the amount, but split into some parts and only after a successful start, not before, when it is really needed.
First Bad Commit
No response
Relevant log output
Logs