ollama can't run qwen:72b, error msg ""gpu VRAM usage didn't recover within timeout

[changingshow]

What is the issue?

I have already downloaded qwen:7b, but when i run ollama run qwen:7b,got this error Error: timed out waiting for llama runner to start:, in the server.log have this msg gpu VRAM usage didn't recover within timeout

OS

Windows

GPU

Nvidia

CPU

Intel

Ollama version

ollama version is 0.1.37

[chrisoutwright]

I also get it (with smaller Params) when running RTX 2080 TI OR GTX 1060 with codeqwen:chat and codegamma:instruct for Win10.
It stays in RAM but will have to copy to GPU RAM everytime after one chat POST.

The GPU RAM is not exceeding on them, so not sure why it timesout every time.

[dhiltgen]

This was fixed in 0.1.38 via pr #4430. If you're still seeing problems after upgrading, please share your server log and I'll reopen.

[pamanseau]

@dhiltgen I have the same issue with 0.1.38 with Linux
ollama.log

[dhiltgen]

@pamanseau from the logs you shared, it looks like the client gave up before the model finished loading, and since the client request was canceled, we canceled the loading of the model. Are you using our CLI, or are you calling the API? If you're calling the API, what timeout are you setting in your client?