Skip to content

Update the llm backend ports #1172

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 22, 2024
Merged

Update the llm backend ports #1172

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ChatQnA/docker_compose/amd/gpu/rocm/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -290,7 +290,7 @@ docker compose up -d
Try the command below to check whether the TGI service is ready.

```bash
docker logs ${CONTAINER_ID} | grep Connected
docker logs chatqna-tgi-server | grep Connected
```

If the service is ready, you will get the response like below.
Expand Down
6 changes: 3 additions & 3 deletions ChatQnA/docker_compose/intel/hpu/gaudi/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -314,7 +314,7 @@ For validation details, please refer to [how-to-validate_service](./how_to_valid
Try the command below to check whether the LLM serving is ready.

```bash
docker logs tgi-service | grep Connected
docker logs tgi-gaudi-server | grep Connected
```

If the service is ready, you will get the response like below.
Expand All @@ -327,15 +327,15 @@ For validation details, please refer to [how-to-validate_service](./how_to_valid

```bash
# TGI service
curl http://${host_ip}:9009/v1/chat/completions \
curl http://${host_ip}:8005/v1/chat/completions \
-X POST \
-d '{"model": ${LLM_MODEL_ID}, "messages": [{"role": "user", "content": "What is Deep Learning?"}], "max_tokens":17}' \
-H 'Content-Type: application/json'
```

```bash
# vLLM Service
curl http://${host_ip}:9009/v1/chat/completions \
curl http://${host_ip}:8007/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": ${LLM_MODEL_ID}, "messages": [{"role": "user", "content": "What is Deep Learning?"}]}'
```
Expand Down
4 changes: 2 additions & 2 deletions ChatQnA/docker_compose/nvidia/gpu/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ docker compose up -d
Try the command below to check whether the TGI service is ready.

```bash
docker logs ${CONTAINER_ID} | grep Connected
docker logs tgi-server | grep Connected
```

If the service is ready, you will get the response like below.
Expand All @@ -285,7 +285,7 @@ docker compose up -d
Then try the `cURL` command below to validate TGI.

```bash
curl http://${host_ip}:9009/v1/chat/completions \
curl http://${host_ip}:8008/v1/chat/completions \
-X POST \
-d '{"model": "Intel/neural-chat-7b-v3-3", "messages": [{"role": "user", "content": "What is Deep Learning?"}], "max_tokens":17}' \
-H 'Content-Type: application/json'
Expand Down
Loading