Cookbook Models for Windows via Docker

[Reien]

Not sure if it's already asked and answered here, but I'm posting it here anyway. If it's already asked, please link me to the correct discussion.

The issue is, I'm having trouble running downloaded models in Cookbook. I already have Ollama set up on my PC and downloaded some models using Powershell, but would also like to download via Cookbook for easier and more convenient access.

Just to give a proper breakdown of what happened:

• I had issues with downloading models with Cookbook
• Created an account in HuggingFace
• Setup a Read-only Token
• Pasted the token in Cookbook > Settings > HF Token Textbox
• Still didn't work
• downloaded llama_cpp in Dependencies
• Manually edited my .env file and added HF_TOKEN=...
• Can finally download models in Cookbook, downloaded llama.cpp models (Qwen2.5-3B-Instruct-GGUF as an example)
• Can't see the downloaded models in Chat and in Odysseus > Settings > Add Models
• Went to Cookbook > Serve
• Clicked the Model I downloaded, see Backend set to llama.cpp
• Clicked Launch
• Get error "Configuring incomplete" in the logs, and a red warning under it
  "llama.cpp found nvcc, but the CUDA runtime library is missing.
  Suggested action: relaunch with the updated runner so llama.cpp builds CPU-only, or install a complete CUDA toolkit/runtime on this server for GPU llama.cpp."

• Serve shows as "ready"
• Model shows up in Chat and Settings > Add Models
• Set it as my Chat/Agent model, and I send a simple "Hello"
• Processing request > Checking model endpoint > Endpoint offline - cancelling

With the help of ChatGPT to troubleshoot, I found that:

• Docker GPU support > Working
• Qwen model file > Working
• llama.cpp backend >Working
• OpenAI API endpoint >Working
• /v1/models >Working
• /v1/chat/completions >Working
• Response format > Valid
• Odysseus reaches endpoint > Working

Along with this suggestion/findings:

The command I'd run next
Inside the container:
grep -R "The model returned an empty response" /app 2>/dev/null

If that finds the source file, we can inspect the exact condition that triggers that message and stop guessing.
At this point, I no longer think this is a configuration issue on your side. The evidence points much more strongly toward a bug or incompatibility in how Odysseus handles responses from its llama.cpp backend.

Not sure how much of that is true, but I've tried a bunch of tests with what ChatGPT recommended, and that's its findings based on the results.
It always tried to steer me away from downloading and installing NVIDIA CUDA stuff for Windows, saying we should avoid it for now just to be sure. So I'm not sure if I'm just running around, or actually resolving the issue at hand.

Would be nice to get help and finally fix this, so I can try using Odysseus more, before trying to use Tailscale.

Thank you!

[meghanaRraj]

I have the same issue, going to troubleshoot now. Will let you know if I find a fix