Why does it takes the same amount of RAM as VRAM?

[LaszloLango]

I noticed something odd recently. I do not have much RAM (24GB), so I try to choose models that fits in my VRAM (16GB). I am on windows. When I load a model that fits in my VRAM it allocates the exact same amount of RAM as well.

llama-server reports that it fits perfectly:

common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected to use 4666 MiB of device memory vs. 15001 MiB of free device memory
llama_params_fit_impl: will leave 10335 >= 1024 MiB of free device memory, no changes needed
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 2.66 seconds

load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
load_tensors: offloading output layer to GPU
load_tensors: offloading 35 repeating layers to GPU
load_tensors: offloaded 37/37 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   394.12 MiB
load_tensors:      Vulkan0 model buffer size =  4076.43 MiB

My first guess is maybe the input file pointer is not freed after loading, but I am not sure. Is this an intended behavior?

The other interesting part is that I do not see that 15,4GB memory allocation on the Processes tab of the Task Manager.

I see the 79% Memory consumption on the Processes tab, but it does not really show what allocates that much memory. The sum of that column does not makes sense. On the other hand on the Performance tab I see it as allocated. Any idea what is happening here? Where is the RAM gone (no pun intended)? Is this normal? Do I always need more RAM then VRAM even if it fits into the VRAM?

[LaszloLango]

#15237 is similar. I also tried --no-mmap and it reduced the RAM usage significantly. Although I still do not understand what the heck is going on. :D

[LaszloLango]

Ok, I think I got it now. RTFM, sorry for the noise.

No Memory Mapping

• --no-mmap: Do not memory-map the model. By default, models are mapped into memory, which allows the system to load only the necessary parts of the model as needed. However, if the model is larger than your total amount of RAM or if your system is low on available memory, using mmap might increase the risk of pageouts, negatively impacting performance. Disabling mmap results in slower load times but may reduce pageouts if you're not using --mlock. Note that if the model is larger than the total amount of RAM, turning off mmap would prevent the model from loading at all.