Error: Ollama call failed with status code 500: llama runner process has terminated

[null4bl3]

After installing using the set-up.sh sript, i installed the mistral model to test it out.

When trying to run a prompt i get the following error message:

Error: Ollama call failed with status code 500: llama runner process has terminated

full error message is here:

web_1        | Error: Ollama call failed with status code 500: llama runner process has terminated
web_1        |     at createOllamaStream (/app/.next/server/chunks/8624.js:2204:23)
web_1        |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
web_1        |     at async ChatOllama._streamResponseChunks (/app/.next/server/chunks/8624.js:2528:26)
web_1        |     at async ChatOllama._call (/app/.next/server/chunks/8624.js:2577:26)
web_1        |     at async ChatOllama._generate (/app/.next/server/chunks/8624.js:6663:22)
web_1        |     at async Promise.allSettled (index 0)
web_1        |     at async ChatOllama.generate (/app/.next/server/chunks/8624.js:6556:25)
web_1        |     at async ChatOllama.call (/app/.next/server/chunks/8624.js:6619:24) {
web_1        |   response: _Response [Response] {
web_1        |     [Symbol(realm)]: null,
web_1        |     [Symbol(state)]: {
web_1        |       aborted: false,
web_1        |       rangeRequested: false,
web_1        |       timingAllowPassed: true,
web_1        |       requestIncludesCredentials: true,
web_1        |       type: 'default',
web_1        |       status: 500,
web_1        |       timingInfo: [Object],
web_1        |       cacheState: '',
web_1        |       statusText: 'Internal Server Error',
web_1        |       headersList: [_HeadersList],
web_1        |       urlList: [Array],
web_1        |       body: [Object]
web_1        |     },
web_1        |     [Symbol(headers)]: _HeadersList {
web_1        |       cookies: null,
web_1        |       [Symbol(headers map)]: [Map],
web_1        |       [Symbol(headers map sorted)]: null
web_1        |     }
web_1        |   }
web_1        | }

[JayNakrani]

Oh, that is weird. Does this happen repeatedly? If so, can you please share the system spec and prompt that you are using? Also looking at the inference container logs will give us a clue as to why llama process died -- may be it ran into OOM error?

[octalxx]

I'm getting the same error.

[GIN] 2023/12/01 - 10:48:37 | 500 |  2.193983447s |    192.168.96.3 | POST     "/api/generate"
2023/12/01 10:48:38 llama.go:292: 8114 MB VRAM available, loading up to 49 GPU layers
2023/12/01 10:48:38 llama.go:421: starting llama runner
2023/12/01 10:48:38 llama.go:479: waiting for llama runner to start responding
2023/12/01 10:48:39 llama.go:436: signal: illegal instruction (core dumped)
2023/12/01 10:48:39 llama.go:444: error starting llama runner: llama runner process has terminated
2023/12/01 10:48:39 llama.go:510: llama runner stopped successfully
2023/12/01 10:48:39 llama.go:421: starting llama runner
2023/12/01 10:48:39 llama.go:479: waiting for llama runner to start responding
2023/12/01 10:48:39 llama.go:436: signal: illegal instruction (core dumped)
2023/12/01 10:48:39 llama.go:444: error starting llama runner: llama runner process has terminated
2023/12/01 10:48:39 llama.go:510: llama runner stopped successfully

This is the output from the inference container.
The prompt is simply a "Hello"
The system is a VM running on Arch with 10cores assigned, 16G of ram and a Nvidia Tesla P4 gpu

[JayNakrani]

@octalxx Thank you for the logs. From my quick search on Ollama discord, I found a similar issue reported by others: ollama/ollama#644

The underlying issue seems to be ggml-org/llama.cpp#1583 -- seems to be a problem with llama.cpp assuming AVX support