Way to disable a gpu from being seen in the kobold.cpp GUI

[wolfwoof88]

Describe the Issue
I have a system with 3 5090 and I'm trying to use the last two for running an AI model while leaves my top gpu open for gaming etc. I have to select all, but then no matter what I pick for my tensor split (0,50,50) for example, the first gpu is still selected for cache/main processing.

I've tried various bat files using cuda hidden to try and hide the gpu from kobold.cpp but to no luck. I know in llama.cpp you can use --devices CUDA1,CUDA2 to fix this. It would be nice if kobold would add something like this to the gui, not just select 1 gpu or all in the dropdown, or some way you can add "extra flags" where I might be able to add commands. that might fix that Sort of like what Oooba/textui has for llama.cpp.

[thijsi123]

On Windows, I have a simple batch file to do this:

@echo off
set CUDA_VISIBLE_DEVICES=1,2
koboldcpp.exe

It also allows me to use the autofit parameter without using my GPU 0.
I'm not sure if it fully disables GPU 0 from being used, but it should.

Would love to see this in the GUI/as a parameter so that I don't always have to load the batch file and the model config manually each time.

[LostRuins]

I'm adding a param --device which allows you to specify exactly what devices get used when ALL is selected E.g.

--device Vulkan0,Vulkan1

follows upstream

[wolfwoof88]

Is that for the command line rather than the GUI? If so thats great, thanks. With my other suggestion is that possible for example I use: --reasoning-budget 0 "command" with llama.cpp / GLM 4.7 since it saves about 5 minutes of unneeded thinking. I'm guessing I might need to use the command line for that.

"On Windows, I have a simple batch file to do this:

@echo off
set CUDA_VISIBLE_DEVICES=1,2
koboldcpp.exe

It also allows me to use the autofit parameter without using my GPU 0.
I'm not sure if it fully disables GPU 0 from being used, but it should.

Would love to see this in the GUI/as a parameter so that I don't always have to load the batch file and the model config manually each time."

I tried something like this but couldn't get it to work, I'll try again thanks for advice.

[Thoron77]

The order of CUDA devices reported for use in Kobold (nvidia-smi) is not always the same as the order of GPU's reported in Windows task manager and even more your first physical GPU you got your monitors hook up to may not be the first one. Having all GPU the same it may be tricky to identify which one you want to keep out of use for AI.
You may need to experiment with CUDA available devices like 0,2 or 0,1 or 1,2 and see which works for you.
Try using nvidia-smi.exe, it is command line tool, with no parameters it will print a table of available GPUs with vRAM usage. Just after booting run it and look which GPU has most vRAM already in use (this should be the CUDA device you want to leave out). But it is not always the case so may still need to experiment with setting available devices.

[henk717]

I think that hiding it in the GUI isn't needed and ill explain why.
The exact thing you want where it doesn't spill into your other GPU's is done automatically internally by us, so unlike other solutions we already do the usual full hiding tricks with CUDA_VISIBLE_DEVICES once you select a GPU in the UI. So yes, in our UI you can see all of them but thats just a visual thing, the library that assigns the VRAM isn't active yet.

Then once you pick the device you do want we hide all the other ones from view unless the all option was chosen. If you do choose the ALL option then its indeed possible to set CUDA_VISIBLE_DEVICES yourself.

That said, lostruins did follow trough and added the --devices option in 1.106.