Served GGUF Model HTTP 404 Error

[ProgramCoin]

Hey everyone,

I've been trying to use a local model I installed from HuggingFace (Fable5) model.

Apparently, the model endpoint is offline but the settings shows it to be there even after I refresh it. So I don't think it would be a download problem, maybe a serve configuration thing?

Everything seemed to go well during the download stage, but after it finished it gives me a message "INFO: : - "GET /v1 HTTP/1.1" 404 Not Found

I hid my IP address and ports in this post, hopefully it's not a problem.
I would appreciate any help or hints on how to resolve this issue.

Thank you in advance!

[darekwojciechowski]

That 404 is a red herring. Nothing serves a bare GET /v1. llama-server only has /v1/models, /v1/chat/completions, /props and /health, so a 404 there is the expected response, not a sign the model failed to load. I'd guess that log line came from you or your browser opening the base URL to test it.

Odysseus doesn't probe /v1 either. It tries /props first, then /v1/models. So test those two:

curl http://<host>:<port>/props
curl http://<host>:<port>/v1/models

If both answer, the server is fine and you have a discovery problem, not a serving one. If they hang or refuse the connection from the machine Odysseus runs on, that's your real bug: llama-server binds to 127.0.0.1 by default, so nothing else can reach it. Restart it with --host 0.0.0.0.

Also check the ports line up: the port in Odysseus settings has to be the one llama-server printed on startup.

[ProgramCoin]

Hey Darek,

Thank you for your response. Very insightful.

I tried running the two commands on Powershell as admin, and I wasn't able to connect to the server.

Based on your response, I believe my next step should be to restart the llama-server binding.
I'm not sure on how to do that. Should I be running the restart command on a specific folder/file and how do I write into the llama-server CLI?

I would also like to ask what the model server process name is called in Odysseus running on a Windows Docker container?
I'm going to try mapping a port to the process because I think it might be a port connection issue now.

[darekwojciechowski]

Ah, Docker changes things. Ignore the --host part of my last reply.

You don't start llama-server by hand. The Cookbook does it for you, and the command it builds already has --host 0.0.0.0 in it.

Your curl failing from PowerShell is expected. In Docker, Cookbook "Local" means inside the Odysseus container, and docker-compose.yml publishes only port 7000, so Windows can't reach the model port. Odysseus doesn't need it to, it talks to the model over the container's own loopback. So don't map a port. Check from the daemon side:

docker compose top odysseus
docker compose exec odysseus curl -s http://localhost:8080/v1/models

Use the port from the endpoint URL in Settings.

If llama-server shows up in top, it's serving fine and the endpoint just never got probed. They're saved with the probe skipped, and Odysseus only re-probes while the Cookbook tab is open. Open the Running panel, find the serve task, and hit "Register endpoint" in its menu.

If top comes back empty, the process died, and the captured output in the Running panel will say why.

Down from 320 to about 190 words. Both of their questions still get answered (no CLI to type into; don't map a port), and the likely root cause — the endpoint never being re-probed — is still the centerpiece. Nothing verified got dropped, only the GPU aside and the procps warning, which was only ever needed to explain a command that's no longer in the answer.

[ProgramCoin]

Thank you.

I ran the two commands in my docker powershell and I got these results:

I believe the second command didnt give me anything, I'm not sure how that works.

How would I be able to re-probe my endpoint? I deleted the model from the serve tab and deleted the downloaded folder in the directory "C:\odysseus\data\huggingface\hub" in hopes I could install this on a clean slate. But now that I re-installed the GGUF model, it shows as finished with no errors but it doesn't show up in the settings. Here's a snip for visual context:

I think the more I try to solve the issue the worse it gets for the both of us 😅. Your help and time is very much appreciated, Darek.