Cannot connect to xinference

[0xchaihu]

Self Checks

• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• Pleas do not modify this template :) and fill in all the required fields.

Dify version

0.5.10

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

Add xinference embedding and rerank models but faiiled.

get xinference model extra parameter failed, url: http://172.21.207.143:9997/v1/models/bge-base-zh, error: HTTPConnectionPool(host='172.21.207.143', port=9997): Max retries exceeded with url: /v1/models/bge-base-zh (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f49d5323070>: Failed to establish a new connection: [Errno 111] Connection refused'))

The IP and the models should be ok, as shown below:

root@LAPTOP-xxxxx:~# hostname -I
172.21.207.143

root@LAPTOP-xxxxx:~# xinference list
UID          Type       Name           Dimensions
-----------  ---------  -----------  ------------
bge-base-zh  embedding  bge-base-zh           768

UID                Type    Name
-----------------  ------  -----------------
bge-reranker-base  rerank  bge-reranker-base

✔️ Expected Behavior

It should work, the models should be able to connect to dify,

❌ Actual Behavior

get xinference model extra parameter failed, url: http://172.21.207.143:9997/v1/models/bge-base-zh, error: HTTPConnectionPool(host='172.21.207.143', port=9997): Max retries exceeded with url: /v1/models/bge-base-zh (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f49d5323070>: Failed to establish a new connection: [Errno 111] Connection refused'))

[dosubot]

Hey @0xchaihu! Great to see you back here. Hope you're doing wonderfully. Let's dive into this issue together.

Based on the information you've provided and the similar issues I found in the repository, it seems like the connection issue you're experiencing might be due to the network configuration between the Dify Docker container and the Xinference service.

In the Docker environment, localhost or 127.0.0.1 refers to the container itself, not the host machine. Therefore, if Xinference is running on your host machine (as indicated by the IP 172.21.207.143), and Dify is running inside a Docker container, Dify will not be able to access Xinference using the localhost address (127.0.0.1) or localhost.

To resolve this issue, you should replace <Machine_IP> in the Server URL http://<Machine_IP>:9997 with the actual IP address of your host machine where Xinference is running. This IP address should be accessible from the Docker container where Dify is deployed. Ensure that this IP is not 127.0.0.1 or localhost but the actual external IP address of the host machine.

Here's a reference from the Dify documentation that explains this: en/tutorials/model-configuration/xinference.md

If you've followed these steps and are still experiencing issues, it could be due to the port mapping in the docker-compose.yaml file. Ensure that you've correctly mapped the container ports to local machine ports. For example, if Xinference is running on port 9997, you should have -p 9997:9997 in your docker-compose.yaml file.

Please let me know if this helps or if you need further assistance.

Sources

• dify cannot connect to xinference LLM
• how to use Xinference's rerank models in Dify ?
• Cannot Add openai-api-compatible model
• en/tutorials/model-configuration/xinference.md

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[crazywoola]

How did you start the xinference client? xinference-local --host 0.0.0.0 --port 9997?

[0xchaihu]

How did you start the xinference client? xinference-local --host 0.0.0.0 --port 9997?

Hi @crazywoola , thanks for your prompt reply.
No, I use xinference-local, I can access http://127.0.0.1:9997/ui. Must use the command you mentioned?

[takatost]

Hey, could you try accessing http://172.21.207.143:9997 on another machine? It might be related to the host firewall rules and it's may set to only allow local access by default.

[0xchaihu]

Hi @crazywoola and @takatost , after using xinference-local --host 0.0.0.0 --port 9997 to start xinference, the models work well as expected. I really appreciate your great support, thank you.