Offload mmproj to RAM and swap with LLM layers to VRAM at runtime to allow more VRAM for the LLM

[Interpause]

Given gpu-layers, I assume there is a mechanism that swaps LLM layers from VRAM to RAM at runtime. Having mmproj for vision capability is useful, but I don't want to have reload models to toggle it, leaving VRAM wasted by mmproj when unused.

Hence I propose:

1. Load mmproj to RAM
2. When user makes request with image, offload later LLM layers to RAM to load mmproj to VRAM
3. Once mmproj is done processing image, concurrently start processing earlier LLM layers while loading the later layers back to VRAM (mmproj goes back to RAM ofc)

Why the above approach?

• Only slightly slower than having mmproj always in VRAM, especially compared to CPU or iGPU offloading.
• Not everyone has more than one GPU.
• Otherwise, the most common system configuration is an AMD iGPU and CUDA dGPU, but that is currently broken:
  • Offloading mmproj to CPU is very slow ofc.
  • Eval bug: mmproj on vulkan extremely degraded compared to cuda for specific images #20081 means offloading mmproj to Vulkan isn't a choice due to quality degradation vs CPU or CUDA.
  • Assuming ROCm doesn't have the same quality degradation problems as Vulkan, ROCm & CUDA cannot co-exist because of Feature Request: mixed ROCm+CUDA possible? #11506.

Alternatives:

• Have a 2nd CUDA GPU.
• Use an Intel GPU with SYCL and hope it doesn't have Vulkan's quality degradation problems.
• Use llama-server built with RPC, and have ROCm run using rpc-server:
  • This incurs unneeded overhead.
  • Given Feature Request: llama-server and llama-cli add --mmproj-device CLI arg #19964, can mmproj even be offloaded to RPC?

EDIT:
I got mmproj offload to RPC ROCm to work using Lemonade's ROCm llama.cpp build. Specifically for my AMD 780M iGPU, I had to set env var HSA_OVERRIDE_GFX_VERSION=11.0.0. Currently testing mmproj speed and if it has the Vulkan quality degradation issue. After testing on the problematic images, ROCm proves to have no numerical errors. But its image processing takes 22s... CPU takes 30s and vulkan around 6s. Yet btop shows that the iGPU is hammered (rather than the CPU, so its not like rpc-server is offloading to CPU). And I specified ROCm0 specifically for the rpc-server, could it be RPC overhead? NVM its dependent on image complexity and how much tokens it resolves to, i think further testing is needed.

[veudal]

This would be huge, since swapping on an image request would be much faster for me too. On CPU, an image request takes about a minute for me at the moment :/