How does offloading work?

[ciori]

Hi,
I am trying to use llama.cpp as a docker container on truenas scale, the host system has an AMD GPU and Vulkan is the only option.

This truenas server is mostly full and has only 1 to 2 GB of free RAM.

My question is: is there a way to run the llama.cpp container so that only the GPU VRAM is used?

I tried with various settings in my docker compose:

    devices:
      - /dev/dri/card0:/dev/dri/card0
      - /dev/dri/renderD128:/dev/dri/renderD128
    command:
      - "--jinja"
      - "-fa"
      - "auto"
      - "-c"
      - "64000"
      - "--no-mmap"
      - "--n-cpu-moe"
      - "0"
      - "-ngl"
      - "999"
      - "--cache-ram"
      - "0"
      - "-m"
      - "/models/gemma-4-E4B-it-UD-Q4_K_XL.gguf"
      - "--api-key"
      - "REDACTED"
      - "--port"
      - "8080"
      - "--host"
      - "0.0.0.0"

and I can see both the CPU and the GPU are recognized from the logs, but I don't realy understand how the offloading is split between them.

The consequence is that when I start the container a VM on truenas gets stopped to free some RAM (the model does work after the VM is stopped and there is enough RAM).

Is there a way to make everything work even on this constrained environment? Also, is there a command similar to ollama ps to view where the model is running and the RAM usage?