Can I keep the models loaded?

[evilblade666]

Hey, folks! I have a question.
Each time I generate a picture with a command-line it starts anew which includes loading a checkpiont, lora(s), etc.. which takes like an extra minute. But is it possible to load all this stuff once and then keep it loaded into memory only taking time for actual image generations?

[stduhpf]

The only solution is to use the server I'm afraid. Then you can send your prompts to the API directly if you don't want to use the webUI.

[evilblade666]

OooKaay.. but I'm afraid that answer didn't make it more clear for me... Can this stable-diffusion.cpp stay loaded as a server of some sorts?..

[stduhpf]

Yes, there are two executables that are included in every build, sd-cli, and sd-server.

[evilblade666]

Oh thank you so much for pushing me in the right direction! Thing is, I've became wondering "is it possible at all to generate on a SteamDeck?", LOL. So I've googled this thread on Reddit: https://www.reddit.com/r/SteamDeck/comments/1hat8fr/hardware_accelerated_ai_image_generation_on_steam/
But the version that a man compiled down there didn't have a server executable at all!
After reading your answer I've downloaded the Linux-Vulkan version of the latest release and IT WORKS!
Tnaks again, like A LOT! :)

[saete]

The flag --mmap keeps the checkpoints, llms and other stuff loaded in the RAM (not the loras) so when you use the cli again it use the memory instead af the ssd or hdd. That's what I understand and use.

[wbruna]

--mmap doesn't do that; you need --params-backend cpu (or the equivalent --offload-to-cpu) - although combining it with --mmap can also help with RAM usage.

[saete]

I thought that was the use of mmap. Maybe because I use --offload-to-cpu I didn't realize that requirement, I just asume of what the cli says:
[INFO ] model_loader.cpp:868 - using mmap for 'models/unet/Qwen-Edit_2511_Q4_K.gguf'
[INFO ] model_loader.cpp:868 - using mmap for models/clip/Qwen2.5-VL-7B-Instruct.Q4_K_M.gguf'
[INFO ] model_loader.cpp:868 - using mmap for 'models/vae/Qwen_Image-VAE.safetensors'

Anyway, the answer to the question is that you can keep the models loaded using --mmap and --offload-to-cpu in the cli version if you have enough RAM. Also you can use the server version, if you don't care lacking the previews function. I use the cli only for the previews while keeping the models loaded in my RAM.

[wbruna]

--mmap does replace explicit reads with copies from the mapped area, in all cases. It's just that most backends cannot use a memory mapped area directly, so the weights usually need to be copied to the device anyway. The main exception is CPU, which can benefit from it both for offloading (--offload-to-cpu) and for generation (--clip-on-cpu, --vae-on-cpu, etc).
--offload-to-cpu does not need --mmap to work, but a memory-mapped area can reuse the file cache memory, and can be discarded by the OS at any time (because the OS can read it back from the disk on demand); so it helps even if you don't have enough RAM to keep the weights in memory.