"could not load model: rpc error: code = Unavailable desc = error reading from server: EOF","type":""

[noblerboy2004]

LocalAI version:

1.25.0

Environment, CPU architecture, OS, and Version:

Intel Xeon E5-2670v2. Window 11. latest

Describe the bug

I followed the guide here:https://docs.flowiseai.com/embeddings/localai-embeddings. But got error when tryin embedding "could not load model: rpc error: code = Unavailable desc = error reading from server: EOF","type":""

To Reproduce

Expected behavior

Logs

huytr@DockerAI MINGW64 ~/OneDrive/Desktop/LocalAI/examples/query_data (master)

$ curl http://localhost:8080/v1/embeddings -H "Content-Type: application/json" -d '{
"input": "Test",
"model": "text-embedding-ada-002"
}'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 198 100 134 100 64 57 27 0:00:02 0:00:02 --:--:-- 85{"error":{"code":500,"message":"could not load model: rpc error: code = Unavailable desc = error reading from server: EOF","type":""}}

Additional context

[Aisuko]

Hi @noblerboy2004 , thanks for your feeding back.

The document on https://docs.flowiseai.com/embeddings/localai-embeddings lacks some details of downloading the model from the gallery. And here you should follow below:

Add GALLERIES to your local environment.

Here is a .env in the root folder of localAI, and you can un-commnet the line GALLERIES='[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}]

Tips

If you get bash: export: url:github:go-skynet/model-gallery/index.yaml}]': not a valid identifier, try the command below in the Linux/macOS(maybe) Terminal.

export GALLERIES='[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}]'

Check if the it set up correctly

And run localAI, if the gallery set up correct, you will not see anything. Otherwise, I can see below,

@Aisuko ➜ /workspaces/LocalAI (master) $ make run
go mod edit -replace github.com/nomic-ai/gpt4all/gpt4all-bindings/golang=/workspaces/LocalAI/gpt4all/gpt4all-bindings/golang
go mod edit -replace github.com/go-skynet/go-ggml-transformers.cpp=/workspaces/LocalAI/go-ggml-transformers
go mod edit -replace github.com/donomii/go-rwkv.cpp=/workspaces/LocalAI/go-rwkv
go mod edit -replace github.com/ggerganov/whisper.cpp=/workspaces/LocalAI/whisper.cpp
go mod edit -replace github.com/go-skynet/go-bert.cpp=/workspaces/LocalAI/go-bert
go mod edit -replace github.com/go-skynet/bloomz.cpp=/workspaces/LocalAI/bloomz
go mod edit -replace github.com/mudler/go-stable-diffusion=/workspaces/LocalAI/go-stable-diffusion
go mod edit -replace github.com/mudler/go-piper=/workspaces/LocalAI/go-piper
go mod edit -replace github.com/mudler/go-ggllm.cpp=/workspaces/LocalAI/go-ggllm
go mod download
touch prepare
CGO_LDFLAGS="" go run ./
1:55AM ERR failed loading galleries: invalid character 'n' looking for beginning of object key string
1:55AM INF Starting LocalAI using 4 threads, with models path: /workspaces/LocalAI/models
1:55AM INF LocalAI version:  ()

Download the model from gallery again

curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{"id": "model-gallery@bert-embeddings"}'

Checking the models folder, and it should include similar like below

@Aisuko ➜ /workspaces/LocalAI (master) $ ls -la models/
total 14212
drwxrwxrwx+  2 codespace root          4096 Sep 22 02:00 .
drwxrwxrwx+ 28 codespace root          4096 Sep 22 01:45 ..
-rw-rw-rw-   1 codespace root             0 Sep 22 01:18 .keep
-rw-rw-rw-   1 codespace codespace 14536015 Sep 22 01:58 bert-MiniLM-L6-v2q4_0.bin
-rw-r--r--   1 codespace codespace      111 Sep 22 01:58 bert-embeddings.yaml
-rw-r--r--   1 codespace codespace      118 Sep 22 02:00 text-embedding-ada-002.yaml

Run the command

@Aisuko ➜ /workspaces/LocalAI (master) $ curl http://localhost:8080/v1/embeddings -H "Content-Type: application/json" -d '{"input": "Test","model": "text-embedding-ada-002"}'
{"object":"list","model":"text-embedding-ada-002","data":[{"embedding":[0.009794451,0.025233647,-0.030652339,0.049976226,0.00060352444,-0.016961304,0.057168704,0.040345185,0.0049206,0.0062199696,0.0112329535,-0.006422204,-0.052636802,0.027765067,-0.011874737,-0.06641837,-0.03504632,-0.023078337,-0.046203017,-0.06723776,0.0415467,-0.022584759,0.018209456,0.05247386,0.0545743,0.054174423,0.0071699973,0.047122065,0.0912816,-0.0067009167,0.0077590174,0.019550426,0.06151075,-0.024220366,-0.007664941,-0.13538025,0.06711042,-0.0287357,043102365,-0.046691485,0.019611616,-0.036851883,-0.02893464,-0.036686588,-0.04995581,-0.012328798,0.0044862507,-0.0057770717,0.0045684106,0.05781866,0.07386673,0.04405747,-0.04252,0.014422377,-0.0035878534,-0.040765695,-0.047999077,-0.031220946,-0.0006110305,0.008372485,0.013731053,0.082618885,0.080272496,0.0022632913,-0.016037494,-0.06123375,0.06307059,-0.013220338,0.020869208,0.030042997,0.11140887,0.13764521,0.018567853,-0.10539311,0.0048948466,0.04123295,0.032958206,-0.051163062,0.015735762,-0.03232726,0.016187007,0.01453379.045940023,0.017109625,0.0025593978,-0.006755319,-0.057964433,-0.044806987,-0.0064437715,-0.06391058,0.022258837,0.003288434,-0.05353514,-0.00067069323,0.022117011,0.033361703,0.6726501,0.014127695,0.0140484655,0.058380667,0.012646213,-0.0086575495,0.011512361,0.021736648,-0.024252865,0.056584492,-4.2065627e-33,0.012902781,-0.09714889,0.037843373,0.093457207,0.07282815,0.0066725793,0.07195803,0.02414018,-0.02952085,-0.013277646,0.014608584,0.06696623,0.083929405,-0.024828196,0.0074623167,-0.07762984,0.049159434,-0.004357168,-0.077711575,-0.063926645,-0.04017431,-0.03981924,0.024196543,0.004066373,-0.057582565,-0.010881982,-0.049290832,0.08750097,-0.025341742,0.01974973,0.0047418983,-0.015407911,0.001951735,0.0020715687,0.0072169844,0.021691801,-0.03408934,-0.05324055,-0.025525,0.063053705,-0.009270767,-0.012982492,0.018980013,-0.079040915,-0.0478784,-0.022035161,0.053105198,0.065044,0.054119006,0.022074318,-0.029071452,0.07902639,-0.02255057,-0.012630121,-0.03982624,-0.12389691,0.07073705,0.053818762,-0.032300014,-0.06735284,0.0033382315,0.023810307,-045538433,-0.01870993,-0.07704327,0.079012826,-0.031281907,0.02178305,-0.0145581355,-0.0022138099,-0.030613845,-0.044252023,-0.09389515,0.0119382115,-0.077033915,-0.048977047,0.04186,4.1751157e-33,-0.049862593,0.041092817,-0.021426447,0.13354272,-0.0058348347,-0.037973218,0.12152487,0.02237059,-0.06747321,0.12199945,0.034916777,-0.042790215,0.07689159,0.0769053,-0.059488386,-0.05583156,-0.0015040624,-0.06760822,0.06904252,0.00094517844,-0.01733727,-0.10127397,0.029499305,0.037161477,-0.04330855,0.030127238,-0.03147561,0.07862891,239468,0.0324741,0.16189496,-0.00097633246,-0.021511294,0.05943427,0.07858113,0.081191175,0.011995542,0.1076277,-0.020478807,0.0827142,0.056920063,-0.029817786,0.06242322,0.07241,-0.006797553,-0.036756437,-0.046707954,-0.0466414,0.015287991,-0.057798613,-0.00941298,-0.07641374,0.0683819,-0.060288154,0.07136964,-0.059040148,-0.050588124,-0.04164462,0.07630.009676021,-0.03591903,-0.03910269,0.04135193,-0.01957466,-0.094523855,0.0081073465,0.0021852443,-0.04244632,-0.074514315,-0.08060708,-0.07060752,0.05702184,-0.048133984,0.059387,0.0077545494,0.044888306,0.0062233256,-0.041222535,-0.063529044,-0.0132918665,-1.4602025e-8,-0.043611955,-0.106504634,0.036152616,0.023948327,-0.020474663,-0.015646355,-0.0275632,0.005399213,-0.00754478,-0.035107613,0.072717324,0.0016576999,-0.07591442,-0.048113957,0.01810086,-0.004108331,0.028773928,-0.07611522,0.023479847,0.027703092,0.036676258,-0.0305,-0.019708978,0.00379691,0.08808157,0.046173863,-0.011677472,-0.09076068,-0.0017467594,0.059464883,0.0004912727,0.022558358,0.04580795,0.054454822,-0.045448568,-0.007723019,0.83387,-0.017431762,-0.03261399,-0.031123897,-0.10039791,-0.06680895,0.034433704,0.0073757926,-0.006804914,0.042208076,-0.017471561,0.029443573,0.03805507,-0.10784109,0.050765432,

[noblerboy2004]

Hi @noblerboy2004 , thanks for your feeding back.

The document on https://docs.flowiseai.com/embeddings/localai-embeddings lacks some details of downloading the model from the gallery. And here you should follow below:

Add GALLERIES to your local environment.

Here is a .env in the root folder of localAI, and you can un-commnet the line GALLERIES='[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}]

Tips

If you get bash: export: url:github:go-skynet/model-gallery/index.yaml}]': not a valid identifier, try the command below in the Linux/macOS(maybe) Terminal.

export GALLERIES='[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}]'

Check if the it set up correctly

And run localAI, if the gallery set up correct, you will not see anything. Otherwise, I can see below,

@Aisuko ➜ /workspaces/LocalAI (master) $ make run
go mod edit -replace github.com/nomic-ai/gpt4all/gpt4all-bindings/golang=/workspaces/LocalAI/gpt4all/gpt4all-bindings/golang
go mod edit -replace github.com/go-skynet/go-ggml-transformers.cpp=/workspaces/LocalAI/go-ggml-transformers
go mod edit -replace github.com/donomii/go-rwkv.cpp=/workspaces/LocalAI/go-rwkv
go mod edit -replace github.com/ggerganov/whisper.cpp=/workspaces/LocalAI/whisper.cpp
go mod edit -replace github.com/go-skynet/go-bert.cpp=/workspaces/LocalAI/go-bert
go mod edit -replace github.com/go-skynet/bloomz.cpp=/workspaces/LocalAI/bloomz
go mod edit -replace github.com/mudler/go-stable-diffusion=/workspaces/LocalAI/go-stable-diffusion
go mod edit -replace github.com/mudler/go-piper=/workspaces/LocalAI/go-piper
go mod edit -replace github.com/mudler/go-ggllm.cpp=/workspaces/LocalAI/go-ggllm
go mod download
touch prepare
CGO_LDFLAGS="" go run ./
1:55AM ERR failed loading galleries: invalid character 'n' looking for beginning of object key string
1:55AM INF Starting LocalAI using 4 threads, with models path: /workspaces/LocalAI/models
1:55AM INF LocalAI version:  ()

Download the model from gallery again

curl $LOCALAI/models/apply -H "Content-Type: application/json" -d '{"id": "model-gallery@bert-embeddings"}'

Checking the models folder, and it should include similar like below

@Aisuko ➜ /workspaces/LocalAI (master) $ ls -la models/
total 14212
drwxrwxrwx+  2 codespace root          4096 Sep 22 02:00 .
drwxrwxrwx+ 28 codespace root          4096 Sep 22 01:45 ..
-rw-rw-rw-   1 codespace root             0 Sep 22 01:18 .keep
-rw-rw-rw-   1 codespace codespace 14536015 Sep 22 01:58 bert-MiniLM-L6-v2q4_0.bin
-rw-r--r--   1 codespace codespace      111 Sep 22 01:58 bert-embeddings.yaml
-rw-r--r--   1 codespace codespace      118 Sep 22 02:00 text-embedding-ada-002.yaml

Run the command

@Aisuko ➜ /workspaces/LocalAI (master) $ curl http://localhost:8080/v1/embeddings -H "Content-Type: application/json" -d '{"input": "Test","model": "text-embedding-ada-002"}'
{"object":"list","model":"text-embedding-ada-002","data":[{"embedding":[0.009794451,0.025233647,-0.030652339,0.049976226,0.00060352444,-0.016961304,0.057168704,0.040345185,0.0049206,0.0062199696,0.0112329535,-0.006422204,-0.052636802,0.027765067,-0.011874737,-0.06641837,-0.03504632,-0.023078337,-0.046203017,-0.06723776,0.0415467,-0.022584759,0.018209456,0.05247386,0.0545743,0.054174423,0.0071699973,0.047122065,0.0912816,-0.0067009167,0.0077590174,0.019550426,0.06151075,-0.024220366,-0.007664941,-0.13538025,0.06711042,-0.0287357,043102365,-0.046691485,0.019611616,-0.036851883,-0.02893464,-0.036686588,-0.04995581,-0.012328798,0.0044862507,-0.0057770717,0.0045684106,0.05781866,0.07386673,0.04405747,-0.04252,0.014422377,-0.0035878534,-0.040765695,-0.047999077,-0.031220946,-0.0006110305,0.008372485,0.013731053,0.082618885,0.080272496,0.0022632913,-0.016037494,-0.06123375,0.06307059,-0.013220338,0.020869208,0.030042997,0.11140887,0.13764521,0.018567853,-0.10539311,0.0048948466,0.04123295,0.032958206,-0.051163062,0.015735762,-0.03232726,0.016187007,0.01453379.045940023,0.017109625,0.0025593978,-0.006755319,-0.057964433,-0.044806987,-0.0064437715,-0.06391058,0.022258837,0.003288434,-0.05353514,-0.00067069323,0.022117011,0.033361703,0.6726501,0.014127695,0.0140484655,0.058380667,0.012646213,-0.0086575495,0.011512361,0.021736648,-0.024252865,0.056584492,-4.2065627e-33,0.012902781,-0.09714889,0.037843373,0.093457207,0.07282815,0.0066725793,0.07195803,0.02414018,-0.02952085,-0.013277646,0.014608584,0.06696623,0.083929405,-0.024828196,0.0074623167,-0.07762984,0.049159434,-0.004357168,-0.077711575,-0.063926645,-0.04017431,-0.03981924,0.024196543,0.004066373,-0.057582565,-0.010881982,-0.049290832,0.08750097,-0.025341742,0.01974973,0.0047418983,-0.015407911,0.001951735,0.0020715687,0.0072169844,0.021691801,-0.03408934,-0.05324055,-0.025525,0.063053705,-0.009270767,-0.012982492,0.018980013,-0.079040915,-0.0478784,-0.022035161,0.053105198,0.065044,0.054119006,0.022074318,-0.029071452,0.07902639,-0.02255057,-0.012630121,-0.03982624,-0.12389691,0.07073705,0.053818762,-0.032300014,-0.06735284,0.0033382315,0.023810307,-045538433,-0.01870993,-0.07704327,0.079012826,-0.031281907,0.02178305,-0.0145581355,-0.0022138099,-0.030613845,-0.044252023,-0.09389515,0.0119382115,-0.077033915,-0.048977047,0.04186,4.1751157e-33,-0.049862593,0.041092817,-0.021426447,0.13354272,-0.0058348347,-0.037973218,0.12152487,0.02237059,-0.06747321,0.12199945,0.034916777,-0.042790215,0.07689159,0.0769053,-0.059488386,-0.05583156,-0.0015040624,-0.06760822,0.06904252,0.00094517844,-0.01733727,-0.10127397,0.029499305,0.037161477,-0.04330855,0.030127238,-0.03147561,0.07862891,239468,0.0324741,0.16189496,-0.00097633246,-0.021511294,0.05943427,0.07858113,0.081191175,0.011995542,0.1076277,-0.020478807,0.0827142,0.056920063,-0.029817786,0.06242322,0.07241,-0.006797553,-0.036756437,-0.046707954,-0.0466414,0.015287991,-0.057798613,-0.00941298,-0.07641374,0.0683819,-0.060288154,0.07136964,-0.059040148,-0.050588124,-0.04164462,0.07630.009676021,-0.03591903,-0.03910269,0.04135193,-0.01957466,-0.094523855,0.0081073465,0.0021852443,-0.04244632,-0.074514315,-0.08060708,-0.07060752,0.05702184,-0.048133984,0.059387,0.0077545494,0.044888306,0.0062233256,-0.041222535,-0.063529044,-0.0132918665,-1.4602025e-8,-0.043611955,-0.106504634,0.036152616,0.023948327,-0.020474663,-0.015646355,-0.0275632,0.005399213,-0.00754478,-0.035107613,0.072717324,0.0016576999,-0.07591442,-0.048113957,0.01810086,-0.004108331,0.028773928,-0.07611522,0.023479847,0.027703092,0.036676258,-0.0305,-0.019708978,0.00379691,0.08808157,0.046173863,-0.011677472,-0.09076068,-0.0017467594,0.059464883,0.0004912727,0.022558358,0.04580795,0.054454822,-0.045448568,-0.007723019,0.83387,-0.017431762,-0.03261399,-0.031123897,-0.10039791,-0.06680895,0.034433704,0.0073757926,-0.006804914,0.042208076,-0.017471561,0.029443573,0.03805507,-0.10784109,0.050765432,

Hi Aisuko,

Thank you for your reply. I tried with your guideline. I'm facing problem.

Would you please help to solve it?

Thank you.

[noblerboy2004]

This is log in execute terminal of docker.

[Aisuko]

Hi, according to the https://localai.io/howtos/easy-setup-full/index.html. It looks like will not work under Git Bash on Windows. I suggest you use docker https://localai.io/howtos/easy-setup-docker-cpu/. If you have GPU environment, check the easy-setup-docker-GPU

[noblerboy2004]

Hi, according to the https://localai.io/howtos/easy-setup-full/index.html. Git bash will not working on Windows. I suggest you use docker https://localai.io/howtos/easy-setup-docker-cpu/. If you have GPU environment, check the easy-setup-docker-GPU

Hi Aisuko,

I did try both option: Bash on windows with linux embedded install and docker. Both show same error.
Then, I tried other model and working normal.

I could be problem with bert model :(

Thank you.

[lunamidori5]

Hi, according to the https://localai.io/howtos/easy-setup-full/index.html. Git bash will not working on Windows. I suggest you use docker https://localai.io/howtos/easy-setup-docker-cpu/. If you have GPU environment, check the easy-setup-docker-GPU

@Aisuko (For the use of the program I made to help with setup, it does not work under git bash, I am working on a fix for that, I am not sure about localai stuff when running locally)

@noblerboy2004 can you post a photo of your nodes from flowise?

[noblerboy2004]

Hi, according to the https://localai.io/howtos/easy-setup-full/index.html. Git bash will not working on Windows. I suggest you use docker https://localai.io/howtos/easy-setup-docker-cpu/. If you have GPU environment, check the easy-setup-docker-GPU

@Aisuko (For the use of the program I made to help with setup, it does not work under git bash, I am working on a fix for that, I am not sure about localai stuff when running locally)

@noblerboy2004 can you post a photo of your nodes from flowise?

Here you are

[Aisuko]

Hi, according to the https://localai.io/howtos/easy-setup-full/index.html. Git bash will not working on Windows. I suggest you use docker https://localai.io/howtos/easy-setup-docker-cpu/. If you have GPU environment, check the easy-setup-docker-GPU

@Aisuko (For the use of the program I made to help with setup, it does not work under git bash, I am working on a fix for that, I am not sure about localai stuff when running locally)

@noblerboy2004 can you post a photo of your nodes from flowise?

Noticed and thanks.

[Aisuko]

Hi, according to the https://localai.io/howtos/easy-setup-full/index.html. Git bash will not working on Windows. I suggest you use docker https://localai.io/howtos/easy-setup-docker-cpu/. If you have GPU environment, check the easy-setup-docker-GPU

Hi Aisuko,

I did try both option: Bash on windows with linux embedded install and docker. Both show same error. Then, I tried other model and working normal.

I could be problem with bert model :(

Thank you.

Do you have some environments like WSL2? Maybe try again in the WSL2 environment.

[lunamidori5]

@noblerboy2004 and @Aisuko I think its the way that flowise does embeds. I can not seem to get them to work on my end too...

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

It seems like there is an issue with the model or the LocalAI installation. Here are some possible solutions to try:

1. Check if the model is compatible with LocalAI by looking at the documentation for the specific model and checking the minimum requirements. You can also use the localai models command to list all available models and their compatibility information.
2. Ensure that the installation of LocalAI is up-to-date by running localai update or reinstalling the container image if necessary.
3. Check the network connectivity between the client and server, as the error suggests a problem with communication between the two. You can try restarting the LocalAI server or checking the network settings on the client.
4. If the issue persists, you can open an issue on the LocalAI GitHub repository with the details of the problem and any logs or additional context that may be helpful in debugging the issue further.

Sources:

• https://github.com/go-skynet/LocalAI/tree/master/.github/ISSUE_TEMPLATE/bug_report.md
• https://localai.io/model-compatibility/exllama/index.html
• https://github.com/go-skynet/LocalAI/tree/master/.env

[noblerboy2004]

Hi, according to the https://localai.io/howtos/easy-setup-full/index.html. Git bash will not working on Windows. I suggest you use docker https://localai.io/howtos/easy-setup-docker-cpu/. If you have GPU environment, check the easy-setup-docker-GPU

Hi Aisuko,
I did try both option: Bash on windows with linux embedded install and docker. Both show same error. Then, I tried other model and working normal.
I could be problem with bert model :(
Thank you.

Do you have some environments like WSL2? Maybe try again in the WSL2 environment.

Hi Aisuko,

The log I provided is on WSL2 Machine. Windows server 2022.

[noblerboy2004]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me, I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

It seems like there is an issue with the model or the LocalAI installation. Here are some possible solutions to try:

1. Check if the model is compatible with LocalAI by looking at the documentation for the specific model and checking the minimum requirements. You can also use the localai models command to list all available models and their compatibility information.
2. Ensure that the installation of LocalAI is up-to-date by running localai update or reinstalling the container image if necessary.
3. Check the network connectivity between the client and server, as the error suggests a problem with communication between the two. You can try restarting the LocalAI server or checking the network settings on the client.
4. If the issue persists, you can open an issue on the LocalAI GitHub repository with the details of the problem and any logs or additional context that may be helpful in debugging the issue further.

Sources:

• https://github.com/go-skynet/LocalAI/tree/master/.github/ISSUE_TEMPLATE/bug_report.md
• https://localai.io/model-compatibility/exllama/index.html
• https://github.com/go-skynet/LocalAI/tree/master/.env

Thanks localai-bot.

I tried as in the picture above and this working now. Thank you for your support.

[Aisuko]

Hi @noblerboy2004 thanks for your feedback. I'm going to close this now.