num_gpu option confusion #603
Description
This isn't really an issue or a bug, but maybe would be helpful to add clarity. I'm running two instances of ollama in Docker Desktop on two separate GPUs, and couldn't figure out for the life of me why, whenever I called those instances from python, it always loaded the majority of an LLM onto my CPU instead of the GPU associated with each container.
The problem in the end was I was attempting to use options when calling the chat endpoint. one of the options is num_gpu, which from the name sounds like the number of GPUs you want it to use. Obviously this is just ignorance on my part for not reading all of the Ollama documentation on what each option does, but num_gpu is actually the number of layers you want to move to the GPU, which in the case of gpt-oss:20b is about 25 layers, so if you have num_gpu=1 assuming that's how many GPUs you want it to use, it's going to put about 1.1GiB of the model on your GPU, and 11.8GiB of the model onto your CPU/memory.
Long story short, as is often the case with software development, I was the problem as well as ignorance, but it would be helpful if the option parameters were documented in a README file somewhere in this repo. or more clearly in Ollama's github repo. but just in case anybody else runs into a similar issue, this at least gives them something to find here.