LocalAI produces error 500 when using QnA with LocalAI embeddings

[gregoryca]

LocalAI version:
quay.io/go-skynet/local-ai:v1.20.0-cublas-cuda12-ffmpeg

Environment, CPU architecture, OS, and Version:
Docker-Compose | Intel I9-8950HK ((x86-64, IA64, and AMD64) | ubuntu-22.04.2 live server amd64

Describe the bug
Error 500 when asking flowise/localAI a question using Local QnA with embeddings

To Reproduce
Create a Conversational Retrieval QA Chain using a txt file, ask it a question after creating and saving the flow.

Expected behavior
Question should be anwered in the popup screen right when testing a flow

Logs
flowise-ai | 2023-07-17 13:43:45 [INFO]: ⬆️ POST /api/v1/internal-prediction/37901ea4-63ad-42ec-a179-566b42bacf0f
local-ai | 1:43PM DBG Request received: {"model":"bert-embeddings","file":"","language":"","response_format":"","size":"","prompt":null,"instruction":"","input":["There was once a hare who was friends with a tortoise. One day, he challenged the tortoise to a","the tortoise to a race. Seeing how slow the tortoise was going, the hare thought he’ll win this","he’ll win this easily. So he took a nap while the tortoise kept on going. When the hare woke up, he","hare woke up, he saw that the tortoise was already at the finish line. Much to his chagrin, the","to his chagrin, the tortoise won the race while he was busy sleeping."],"stop":null,"messages":null,"stream":false,"echo":false,"top_p":0,"top_k":0,"temperature":0,"max_tokens":0,"n":0,"batch":0,"f16":false,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"mirostat_eta":0,"mirostat_tau":0,"mirostat":0,"frequency_penalty":0,"tfz":0,"seed":0,"mode":0,"step":0,"typical_p":0}
local-ai | 1:43PM DBG Parameter Config: &{OpenAIRequest:{Model: File: Language: ResponseFormat: Size: Prompt: Instruction: Input: Stop: Messages:[] Stream:false Echo:false TopP:0 TopK:0 Temperature:0 Maxtokens:0 N:0 Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 Seed:0 Mode:0 Step:0 TypicalP:0} Name:bert-embeddings StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:0 F16:false NUMA:false Threads:6 Debug:true Roles:map[] Embeddings:false Backend: TemplateConfig:{Completion: Chat: Edit:} MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:0 MMap:false MMlock:false LowVRAM:false TensorSplit: MainGPU: ImageGenerationAssets: PromptCachePath: PromptCacheAll:false PromptCacheRO:false PromptStrings:[] InputStrings:[There was once a hare who was friends with a tortoise. One day, he challenged the tortoise to a the tortoise to a race. Seeing how slow the tortoise was going, the hare thought he’ll win this he’ll win this easily. So he took a nap while the tortoise kept on going. When the hare woke up, he hare woke up, he saw that the tortoise was already at the finish line. Much to his chagrin, the to his chagrin, the tortoise won the race while he was busy sleeping.] InputToken:[]}
local-ai | [172.22.0.1]:44904 500 - POST /embeddings
local-ai | [172.22.0.1]:57214 500 - POST /embeddings
local-ai | 1:43PM DBG Request received: {"model":"bert-embeddings","file":"","language":"","response_format":"","size":"","prompt":null,"instruction":"","input":["There was once a hare who was friends with a tortoise. One day, he challenged the tortoise to a","the tortoise to a race. Seeing how slow the tortoise was going, the hare thought he’ll win this","he’ll win this easily. So he took a nap while the tortoise kept on going. When the hare woke up, he","hare woke up, he saw that the tortoise was already at the finish line. Much to his chagrin, the","to his chagrin, the tortoise won the race while he was busy sleeping."],"stop":null,"messages":null,"stream":false,"echo":false,"top_p":0,"top_k":0,"temperature":0,"max_tokens":0,"n":0,"batch":0,"f16":false,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"mirostat_eta":0,"mirostat_tau":0,"mirostat":0,"frequency_penalty":0,"tfz":0,"seed":0,"mode":0,"step":0,"typical_p":0}
local-ai | 1:43PM DBG Parameter Config: &{OpenAIRequest:{Model: File: Language: ResponseFormat: Size: Prompt: Instruction: Input: Stop: Messages:[] Stream:false Echo:false TopP:0 TopK:0 Temperature:0 Maxtokens:0 N:0 Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 Seed:0 Mode:0 Step:0 TypicalP:0} Name:bert-embeddings StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:0 F16:false NUMA:false Threads:6 Debug:true Roles:map[] Embeddings:false Backend: TemplateConfig:{Completion: Chat: Edit:} MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:0 MMap:false MMlock:false LowVRAM:false TensorSplit: MainGPU: ImageGenerationAssets: PromptCachePath: PromptCacheAll:false PromptCacheRO:false PromptStrings:[] InputStrings:[There was once a hare who was friends with a tortoise. One day, he challenged the tortoise to a the tortoise to a race. Seeing how slow the tortoise was going, the hare thought he’ll win this he’ll win this easily. So he took a nap while the tortoise kept on going. When the hare woke up, he hare woke up, he saw that the tortoise was already at the finish line. Much to his chagrin, the to his chagrin, the tortoise won the race while he was busy sleeping.] InputToken:[]}
local-ai | 1:43PM DBG Request received: {"model":"bert-embeddings","file":"","language":"","response_format":"","size":"","prompt":null,"instruction":"","input":["There was once a hare who was friends with a tortoise. One day, he challenged the tortoise to a","the tortoise to a race. Seeing how slow the tortoise was going, the hare thought he’ll win this","he’ll win this easily. So he took a nap while the tortoise kept on going. When the hare woke up, he","hare woke up, he saw that the tortoise was already at the finish line. Much to his chagrin, the","to his chagrin, the tortoise won the race while he was busy sleeping."],"stop":null,"messages":null,"stream":false,"echo":false,"top_p":0,"top_k":0,"temperature":0,"max_tokens":0,"n":0,"batch":0,"f16":false,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"mirostat_eta":0,"mirostat_tau":0,"mirostat":0,"frequency_penalty":0,"tfz":0,"seed":0,"mode":0,"step":0,"typical_p":0}
local-ai | [172.22.0.1]:57216 500 - POST /embeddings
local-ai | 1:43PM DBG Parameter Config: &{OpenAIRequest:{Model: File: Language: ResponseFormat: Size: Prompt: Instruction: Input: Stop: Messages:[] Stream:false Echo:false TopP:0 TopK:0 Temperature:0 Maxtokens:0 N:0 Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 Seed:0 Mode:0 Step:0 TypicalP:0} Name:bert-embeddings StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:0 F16:false NUMA:false Threads:6 Debug:true Roles:map[] Embeddings:false Backend: TemplateConfig:{Completion: Chat: Edit:} MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:0 MMap:false MMlock:false LowVRAM:false TensorSplit: MainGPU: ImageGenerationAssets: PromptCachePath: PromptCacheAll:false PromptCacheRO:false PromptStrings:[] InputStrings:[There was once a hare who was friends with a tortoise. One day, he challenged the tortoise to a the tortoise to a race. Seeing how slow the tortoise was going, the hare thought he’ll win this he’ll win this easily. So he took a nap while the tortoise kept on going. When the hare woke up, he hare woke up, he saw that the tortoise was already at the finish line. Much to his chagrin, the to his chagrin, the tortoise won the race while he was busy sleeping.] InputToken:[]}
local-ai | [127.0.0.1]:40870 200 - GET /readyz
local-ai | 1:43PM DBG Request received: {"model":"bert-embeddings","file":"","language":"","response_format":"","size":"","prompt":null,"instruction":"","input":["There was once a hare who was friends with a tortoise. One day, he challenged the tortoise to a","the tortoise to a race. Seeing how slow the tortoise was going, the hare thought he’ll win this","he’ll win this easily. So he took a nap while the tortoise kept on going. When the hare woke up, he","hare woke up, he saw that the tortoise was already at the finish line. Much to his chagrin, the","to his chagrin, the tortoise won the race while he was busy sleeping."],"stop":null,"messages":null,"stream":false,"echo":false,"top_p":0,"top_k":0,"temperature":0,"max_tokens":0,"n":0,"batch":0,"f16":false,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"mirostat_eta":0,"mirostat_tau":0,"mirostat":0,"frequency_penalty":0,"tfz":0,"seed":0,"mode":0,"step":0,"typical_p":0}
local-ai | 1:43PM DBG Parameter Config: &{OpenAIRequest:{Model: File: Language: ResponseFormat: Size: Prompt: Instruction: Input: Stop: Messages:[] Stream:false Echo:false TopP:0 TopK:0 Temperature:0 Maxtokens:0 N:0 Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 Seed:0 Mode:0 Step:0 TypicalP:0} Name:bert-embeddings StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:0 F16:false NUMA:false Threads:6 Debug:true Roles:map[] Embeddings:false Backend: TemplateConfig:{Completion: Chat: Edit:} MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:0 MMap:false MMlock:false LowVRAM:false TensorSplit: MainGPU: ImageGenerationAssets: PromptCachePath: PromptCacheAll:false PromptCacheRO:false PromptStrings:[] InputStrings:[There was once a hare who was friends with a tortoise. One day, he challenged the tortoise to a the tortoise to a race. Seeing how slow the tortoise was going, the hare thought he’ll win this he’ll win this easily. So he took a nap while the tortoise kept on going. When the hare woke up, he hare woke up, he saw that the tortoise was already at the finish line. Much to his chagrin, the to his chagrin, the tortoise won the race while he was busy sleeping.] InputToken:[]}
local-ai | [172.22.0.1]:44582 500 - POST /embeddings
local-ai | 1:44PM DBG Request received: {"model":"bert-embeddings","file":"","language":"","response_format":"","size":"","prompt":null,"instruction":"","input":["There was once a hare who was friends with a tortoise. One day, he challenged the tortoise to a","the tortoise to a race. Seeing how slow the tortoise was going, the hare thought he’ll win this","he’ll win this easily. So he took a nap while the tortoise kept on going. When the hare woke up, he","hare woke up, he saw that the tortoise was already at the finish line. Much to his chagrin, the","to his chagrin, the tortoise won the race while he was busy sleeping."],"stop":null,"messages":null,"stream":false,"echo":false,"top_p":0,"top_k":0,"temperature":0,"max_tokens":0,"n":0,"batch":0,"f16":false,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"mirostat_eta":0,"mirostat_tau":0,"mirostat":0,"frequency_penalty":0,"tfz":0,"seed":0,"mode":0,"step":0,"typical_p":0}
local-ai | [172.22.0.1]:51836 500 - POST /embeddings
local-ai | 1:44PM DBG Parameter Config: &{OpenAIRequest:{Model: File: Language: ResponseFormat: Size: Prompt: Instruction: Input: Stop: Messages:[] Stream:false Echo:false TopP:0 TopK:0 Temperature:0 Maxtokens:0 N:0 Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 Seed:0 Mode:0 Step:0 TypicalP:0} Name:bert-embeddings StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:0 F16:false NUMA:false Threads:6 Debug:true Roles:map[] Embeddings:false Backend: TemplateConfig:{Completion: Chat: Edit:} MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:0 MMap:false MMlock:false LowVRAM:false TensorSplit: MainGPU: ImageGenerationAssets: PromptCachePath: PromptCacheAll:false PromptCacheRO:false PromptStrings:[] InputStrings:[There was once a hare who was friends with a tortoise. One day, he challenged the tortoise to a the tortoise to a race. Seeing how slow the tortoise was going, the hare thought he’ll win this he’ll win this easily. So he took a nap while the tortoise kept on going. When the hare woke up, he hare woke up, he saw that the tortoise was already at the finish line. Much to his chagrin, the to his chagrin, the tortoise won the race while he was busy sleeping.] InputToken:[]}
local-ai | 1:44PM DBG Request received: {"model":"bert-embeddings","file":"","language":"","response_format":"","size":"","prompt":null,"instruction":"","input":["There was once a hare who was friends with a tortoise. One day, he challenged the tortoise to a","the tortoise to a race. Seeing how slow the tortoise was going, the hare thought he’ll win this","he’ll win this easily. So he took a nap while the tortoise kept on going. When the hare woke up, he","hare woke up, he saw that the tortoise was already at the finish line. Much to his chagrin, the","to his chagrin, the tortoise won the race while he was busy sleeping."],"stop":null,"messages":null,"stream":false,"echo":false,"top_p":0,"top_k":0,"temperature":0,"max_tokens":0,"n":0,"batch":0,"f16":false,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"mirostat_eta":0,"mirostat_tau":0,"mirostat":0,"frequency_penalty":0,"tfz":0,"seed":0,"mode":0,"step":0,"typical_p":0}
local-ai | 1:44PM DBG Parameter Config: &{OpenAIRequest:{Model: File: Language: ResponseFormat: Size: Prompt: Instruction: Input: Stop: Messages:[] Stream:false Echo:false TopP:0 TopK:0 Temperature:0 Maxtokens:0 N:0 Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 Seed:0 Mode:0 Step:0 TypicalP:0} Name:bert-embeddings StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:0 F16:false NUMA:false Threads:6 Debug:true Roles:map[] Embeddings:false Backend: TemplateConfig:{Completion: Chat: Edit:} MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:0 MMap:false MMlock:false LowVRAM:false TensorSplit: MainGPU: ImageGenerationAssets: PromptCachePath: PromptCacheAll:false PromptCacheRO:false PromptStrings:[] InputStrings:[There was once a hare who was friends with a tortoise. One day, he challenged the tortoise to a the tortoise to a race. Seeing how slow the tortoise was going, the hare thought he’ll win this he’ll win this easily. So he took a nap while the tortoise kept on going. When the hare woke up, he hare woke up, he saw that the tortoise was already at the finish line. Much to his chagrin, the to his chagrin, the tortoise won the race while he was busy sleeping.] InputToken:[]}
local-ai | [172.22.0.1]:55930 500 - POST /embeddings
local-ai | [127.0.0.1]:43450 200 - GET /readyz
local-ai | 1:45PM DBG Request received: {"model":"bert-embeddings","file":"","language":"","response_format":"","size":"","prompt":null,"instruction":"","input":["There was once a hare who was friends with a tortoise. One day, he challenged the tortoise to a","the tortoise to a race. Seeing how slow the tortoise was going, the hare thought he’ll win this","he’ll win this easily. So he took a nap while the tortoise kept on going. When the hare woke up, he","hare woke up, he saw that the tortoise was already at the finish line. Much to his chagrin, the","to his chagrin, the tortoise won the race while he was busy sleeping."],"stop":null,"messages":null,"stream":false,"echo":false,"top_p":0,"top_k":0,"temperature":0,"max_tokens":0,"n":0,"batch":0,"f16":false,"ignore_eos":false,"repeat_penalty":0,"n_keep":0,"mirostat_eta":0,"mirostat_tau":0,"mirostat":0,"frequency_penalty":0,"tfz":0,"seed":0,"mode":0,"step":0,"typical_p":0}
local-ai | 1:45PM DBG Parameter Config: &{OpenAIRequest:{Model: File: Language: ResponseFormat: Size: Prompt: Instruction: Input: Stop: Messages:[] Stream:false Echo:false TopP:0 TopK:0 Temperature:0 Maxtokens:0 N:0 Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 Seed:0 Mode:0 Step:0 TypicalP:0} Name:bert-embeddings StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:0 F16:false NUMA:false Threads:6 Debug:true Roles:map[] Embeddings:false Backend: TemplateConfig:{Completion: Chat: Edit:} MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:0 MMap:false MMlock:false LowVRAM:false TensorSplit: MainGPU: ImageGenerationAssets: PromptCachePath: PromptCacheAll:false PromptCacheRO:false PromptStrings:[] InputStrings:[There was once a hare who was friends with a tortoise. One day, he challenged the tortoise to a the tortoise to a race. Seeing how slow the tortoise was going, the hare thought he’ll win this he’ll win this easily. So he took a nap while the tortoise kept on going. When the hare woke up, he hare woke up, he saw that the tortoise was already at the finish line. Much to his chagrin, the to his chagrin, the tortoise won the race while he was busy sleeping.] InputToken:[]}
local-ai | [172.22.0.1]:53326 500 - POST /embeddings
flowise-ai | 2023-07-17 13:45:10 [ERROR]: Request failed with status code 500
flowise-ai | Error: Request failed with status code 500
flowise-ai | at createError (/usr/local/lib/node_modules/flowise/node_modules/openai/node_modules/axios/lib/core/createError.js:16:15)
flowise-ai | at settle (/usr/local/lib/node_modules/flowise/node_modules/openai/node_modules/axios/lib/core/settle.js:17:12)
flowise-ai | at IncomingMessage.handleStreamEnd (/usr/local/lib/node_modules/flowise/node_modules/openai/node_modules/axios/lib/adapters/http.js:322:11)
flowise-ai | at IncomingMessage.emit (node:events:525:35)
flowise-ai | at endReadableNT (node:internal/streams/readable:1359:12)
flowise-ai | at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
flowise-ai | 2023-07-17 13:45:10 [ERROR]: [server] [mode:child]: Error: Error: Request failed with status code 500
flowise-ai | Error: Error: Request failed with status code 500
flowise-ai | at buildLangchain (/usr/local/lib/node_modules/flowise/dist/utils/index.js:227:19)
flowise-ai | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
flowise-ai | at async ChildProcess.runChildProcess (/usr/local/lib/node_modules/flowise/dist/ChildProcess.js:107:40)
flowise-ai | at async process. (/usr/local/lib/node_modules/flowise/dist/ChildProcess.js:177:9)

Additional context
I have attached my flow so you won't have to build it yourself when testing -
Document Chatbot - LocalAI Chatflow.zip

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

I am sorry that you are having trouble with your hare and tortoise story, it seems like the hare didn't take into account his opponent's capabilities before challenging him to a race. I think it's important to remember that sometimes slow and steady wins the race, and it's always better to focus on consistent progress rather than quick fixes. It is also important to pay attention to your surroundings and adapt to the situation you are in, so that you can win the game you started playing.

Regarding the technical issue with your Github assistant, I see that there was a 500 error when the flowise-ai tried to call the GET /embeddings endpoint of the local-ai server. It looks like there might be some missing configuration options for this request or something is not correctly installed on your end, causing this error. Can you please provide more information about what version of the LocalAI framework you are using and any other details that may help to resolve this issue?

Sources:

• https://github.com/go-skynet/LocalAI/tree/master/examples/chatbot-ui-manual/docker-compose.yaml
• https://github.com/go-skynet/LocalAI/tree/master/api/openai/chat.go
• https://github.com/go-skynet/LocalAI/tree/master/.github/ISSUE_TEMPLATE/bug_report.md

[gregoryca]

Everything i am using is the latest version except my GPU driver, as it will crash it cannot find CUDA.
My NVCC version:
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Mon_Oct_24_19:12:58_PDT_2022
Cuda compilation tools, release 12.0, V12.0.76
Build cuda_12.0.r12.0/compiler.31968024_0

My Nvidia-SMI cuda version:
NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2

I am using traefik i.c.m. dynamic config file which looks like the following:
HTTP router:
localai-secure:
service: localai
rule: Host(ai.apex-migrations.net)
entryPoints:
- web-secure
middlewares:
- real-ip-foo@file
- IP2Location@file
- requestHeader@file
tls:
certResolver: production

Middlewares:
flowwise-secure:
service: flowwise
rule: Host(localai.apex-migrations.net)
entryPoints:
- web-secure
middlewares:
- real-ip-foo@file
- IP2Location@file
- requestHeader@file
tls:
certResolver: production

Services:
localai:
loadBalancer:
servers:
- url: http://localai.html:80
passHostHeader: false

flowwise:
  loadBalancer:
    servers:
      - url: http://192.168.redacted:3000
    passHostHeader: true

Both containers run on a different machine where traefik runs from, but i can route the traffic from my main server/traefik container to the rest of my network.

I could try to run a trace of what happends on the server when it tries to send the request where it receives a error 500

[gregoryca]

I have to add to this, the flow only crashes when i upload a PDF. The same flow works flawless when i use a TXT file.

Could it something in the PDF itself ?
I can expose an endpoint with the TXT option, and another endpoint with Dozzle Logs, so you can test it out yourself when you want.

[gregoryca]

Bug is fixed in latest version.

[sandriansandy]

Im using the latest version but got the same error log messages

whats happen? and how to fix it?

[hwamil]

Also getting this on Apple M2