M-series Macs running llama.cpp in GPU-Accelerated Containers - New Benchmark Results #12985
AndreasKunar
started this conversation in
Show and tell
Replies: 9 comments 19 replies
That's because Flash Attention in Vulkan is currently only implemented for Nvidia GPUs with coopmat2, on any other device it falls back to CPU, which will slow you down a lot.
That is because Vulkan still accelerates large matrix multiplications that are slow on CPU, by transferring the data to GPU and calculating it there. Edit: Also great writeup, very interesting. I'm happy that Vulkan is useful on Mac, too, despite being slower than Metal. |
0 replies
|
I have a base model M1 Macbook Air I decided to test. This only has 8GB of RAM so it can only run a 1B model. Bare metal CPUBare Metal GPUlocal/fedora-krunkit CPUlocal/fedora-krunkit GPURamaLama provides a prebuilt image based on RamaLama CPURamaLama GPU |
1 reply
|
Cool, thanks a lot! The comparison with containers/RamaLama is very cool, I did not know them, and that they support Podman+krunkit Apparently the use of the Kompute llama.cpp-backend is a problem in your example, not sure if there is a Vulkan-RamaLama container with Fedora40+krunkit-mesa-driver. If there is, it should have the same, faster performance. Also, to me it's totally strange, that gemma3 1B Q4_0 TG on the bare metal CPU (-ngl 0) is faster than when using the bare metal GPU (-ngl 99). Its similar on my M4 Pro (I seem to have a slightly different model - google/gemma-3-1b-it-qat-q4_0-gguf): -ngl 0
-ngl 99
|
|
Yeah weird that running on CPU seems to have faster text generation. I noticed this before as well while using whisper.cpp. Looking at I think the reason |
2 replies
|
I tried to run RamaLama on my M2 Max to get comparable numbers for RamaLama(Kompute) vs. my initial Fedora40/mesa-krunkit/Vulkan, but I get very strange "memoryErrorOutOfDeviceMemory" errors on my 60GB 8-CPU (!) Podman krunkit machine when it tries to offload all llama 2 7B layers to the GPU with Kompute (it worked with my self-built container and Vulkan). Also gemma 3 1B is quite small and maybe an unusual benchmark, here FYI my two systems' bare-metal data with gemma 3 1b and old llama 2 7b. The M4 generation seems to consume a bit less power for a similar result, and its CPUs provide better performance, even with its slower RAM-bandwidth. Also, the GPU is not only usually faster, but also more energy-efficient per token than the CPU.
|
|
Those are some interesting results. I have no idea why RamaLama decided to build with Kompute over Vulkan which I didn't even know was a thing before this. I would run the benchmark with |
|
I was checking out the |
1 reply
|
Here is a quick containerfile I wrote to build llama.cpp and install Mesa krunkit driver using AlmaLinux 9 . |
|
I remembered there is a fork of Ollama with Vulkan support. It's an old build so doesn't support new models like Gemma 3. I had to patch the |
1 reply
|
What did you do to patch the gpu.go? |
|
It would be really interesting to get some performance comparisons with the same models running under Docker Model Runner |
3 replies
|
I assume Docker Model Runner will have the same performance as running llama.cpp on bare metal since that's what it does. |
|
Thanks for pointing this out, I did not know Docker Model Runner - its not virtualizing the local GPU(s) or Vulkan, instead just providing a closed-source, open-AI compatible, llama.cpp based LLM-API to Docker-containers, and a convenient access to a curated model-store. It will likely have native llama.cpp performance - but with Docker's licensing, no enhanced security. And in my opinion not any more convenience than e.g. just brew-installing and running local ollama - which is open-source , not bound to Docker, and has a more permissive licensing. Or you could just run a plain llama.cpp llama-server with maybe a tiny bit less convenience, but even more flexibility. Docker Model Runner seems just focused on simplification/convenience. A bit like ollama or other company efforts like e.g. NVIDIA's AI Playground or Microsoft's AI Toolkit for Visual Studio Code. It just integrates some kind of llama.cpp llama-server derivative into the host docker-desktop app, together with a kind of curated model-store, from which you can pull your models.
P.S.: What I forgot to highlight on Podman+krunkit is, that it runs rootless. I experiment in parallel with NVIDIA-GPU container access. E.g. on NVIDIA Jetson's Jetpack, which has Docker pre-installed, GPU-access from docker-containers apparently cannot run in a rootless Docker. At least I was not able to find anybody on the internet who managed this successfully - I killed my Jetson installation two times while trying, and had to completely wipe+reinstall it. |
|
Just a clarification on Docker Model Runner (DMR). I had a conversation with someone from Docker and he explained their reasoning. I think I understand their motivation now better and see the following advantages of DMR, which look promising:
P.S. he also explained, that Vulkan-redirect out of Docker is also technically possible, albeit with the known technical heavy-lifting and performance-penalties. Apparently this is why they went the DMR path, and I now think, that I understand the why much better. |
|
I randomly found this script which references a |
1 reply
|
I now got more information on Docker Model Runner (DMR) and it seems a much more interesting concept than I thought - Virtualizing an openAI API with OCI models instead of Vulkan and making this part of corporate-approved Docker. |
|
This might be a bit off topic but has anyone tried to run the Vulkan implementation in a Mac VM rather than a container as done here? The Github CI has Mac VMs available to us and it would be nice if we could run the Vulkan tests on a real GPU as part of the CI. Right now we use Llvmpipe CPU emulation for that and it's super slow. |
7 replies
|
macOS large runners says they have 8 GPU hardware acceleration but also so seems like conflicting information https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners/about-larger-runners#about-macos-larger-runners |
|
For fun I went and tried this out on the CI. It's pretty easy to set up and compiles fine but unfortunately the backend isn't even able to allocate memory so... yeah. |
|
"No suitable memory type found" is unusual. Can you run vulkaninfo on that device @netrunnereve ? |
vulkaninfoThis is with #15905 so the error message is different but still unhelpful. From a quick search I couldn't find anyone complaining about MoltenVK not working with the Apple Paravirtual GPU, but I don't see anyone saying that it works either. |
|
Hmm, yeah, the memory type thing was just due to the error message. I don't know why it is failing. |
M5 Max 128GB — Qwen3.6-27B Q8_0 long-context Metal benchmarkAdding a bare-metal Apple M5 Max result, focusing on prompt processing and token generation performance as context depth increases. Hardware
Model
Configuration
Command./llama-bench -m ./Qwen3.6-27B-Q8_0.gguf -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -p 512 -n 256 -d 0,8192,16384,32768,65536,98304,131072,196608,262144 -r 3Results
Context scalingTG drops from 17.46 t/s at depth 0 to:
There is a particularly noticeable drop between 16K and 32K. PP512 also decreases substantially with context depth, from about 799 t/s at depth 0 to 27 t/s at 256K. The model fits comfortably within unified memory, so I am curious how much of this long-context degradation is expected from attention/KV-cache bandwidth, and how much may be specific to the current Metal implementation or this Qwen architecture. It would be interesting to compare this against other M5 Max results, especially with different KV cache types or recent llama.cpp builds. |
3 replies
|
Don't quantize the cache with Metal - it's not optimized yet. |
|
Thanks — that makes sense. I was using I'll rerun the exact same depth sweep with |
|
I ended up doing a quick partial rerun sooner than expected, but I don't plan to complete the full depth sweep for now. Preliminary results with non-quantized KV cache:
So far, |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Containers provide an important security-perimeter for running less-trusted software. However until now, this has not been quite feasible for Apple-Silicon Macs and llama.cpp. Mac GPUs couldn't really be used in containers. I re-visited the current status of Podman, which has macOS GPU-remoting - here are my benchmark-results and a how-to.
Technology: Podman enables Vulkan API-calls inside its containers to be accelerated via transferring these calls to the host's Apple M-series GPU.
TLDR: Benefits/Caveats GPU-Containers with this are faster than pure CPU containers in Docker, etc. - but the overhead vs. running llama.cpp directly is still significant. In my M2 Max based tests they are approx. 40% slower. But the GPU-container is still 3x faster than a pure CPU-container in prompt-processing, and 25% faster in token-generation. However, building these containers is a bit complicated - therefore below my how-to.
We need the following components:
You find more technical details here:
Installing Podman,...
We need Podman,... installed on MacOS, e.g. with:
Then create the Podman machine as documented here.
You can test your Podman GPU-Container-ready installation with the command
podman run --rm -it --device /dev/dri --name gpu-info quay.io/slopezpa/fedora-vgpu vulkaninfo | grep "GPU"This should display the information about the GPU.
How to Build a GPU-accelerated Container Base Image
For our experiments we need a "Containerfile" which defines what OS and base-software should be installed. The Podman Containerfile (e.g.
fedora-krunkit.containerfile):Then we use Podman to create a container-image from this (best done in an empty directory):
You can test this image with the command
podman run --rm -it --device /dev/dri --name gpu-info local/fedora-krunkit vulkaninfo | grep "GPU"This should again display the information about the GPU.
Playing with llama.cpp and this Base Image
We can now use this image, to start a container and install the toolset, git-clone llama.cpp, build and run it. I suggest to store the large model-files ion the host and share this folder with your container.
podman run --rm -it --device /dev/dri --name llama-cpp-dev -v <your host's model folder>:/models local/fedora-krunkit /bin/bashstarts the container and puts you into it's terminal. Please be careful, if you exit this, you discard all the changes you made. This is good for experimentation, because we can just start again from scratch, if things go wrong. For permanent use, we can build a llama.cpp server or cli container later with our experience.Installing the toolset, llama.cpp and building the server, cli, and bench:
Playing with llama.cpp in that container - e.g. benchmarking - requires a model from the host. You can then run this e.g. with
./build/bin/llama-bench -p 512 -n 128 -ngl 99 -m <your model's .gguf-file>Happy experimenting! My benchmark results are below.
llama.cpp Performance Results
All benchmarking was done on Apple Silicon Macs, with llama-bench and llama-2 Q4_0. This is in order to be comparable to the llama.cpp perfocmance-discussion results), but with the latest llama.cpp-build: b43d89e (5143).
M2 Max Podman Container Performance
On a M2 Max Mac Studio, 8 CPUs / 64GB RAM allocated to the Podman machine.
Container, llama.cpp CPU-only:
Built for the CPU-Backend, without special parameters.
Container with llama.cpp, built for Vulkan, with GPU-remoting:
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Virtio-GPU Venus (Apple M2 Max) (venus) | uma: 1 | fp16: 1 | warp size: 32 | shared memory: 32768 | int dot: 0 | matrix cores: none
Strangely -fa with the Vulkan-backend leads to much worse performance, especially in TG:
Also, building with Vulkan and running with -ngl 0 strangely is faster than the CPU-only container and CPU-native:
For Comparison a "Native" M2 Max:
With GPU (-ngl 99):
Without GPU (-ngl 0, it's similar to the container):
I wrote an earlier, now a bit outdated medium.com article and discussion-item #8042 on this. But thought I should give it another try with the latest versions.
I hope this is useful information - comments and feedback very welcome!
All reactions