Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update docker docs for VL api #1534

Merged
merged 4 commits into from
May 6, 2024
Merged

Conversation

vody-am
Copy link
Contributor

@vody-am vody-am commented Apr 30, 2024

Docs update in relation to issue #1514

Motivation

Updating documentation around Docker images for new users.
It should be made clear to users what is expected, and a way to resolve issues described.

Modification

Adding a warning about dependencies and an example of how to resolve by creating a new Docker image.

@lvhan028
Copy link
Collaborator

@vody-am If you are familiar with Chinese, could you please update the user guide in docs/zh_cn/serving/api_server_vl.md?

@lvhan028 lvhan028 requested a review from AllentDan April 30, 2024 16:09
@lvhan028 lvhan028 added the documentation Improvements or additions to documentation label Apr 30, 2024
@vody-am
Copy link
Contributor Author

vody-am commented Apr 30, 2024

@lvhan028 Google translate has been working well, so yes I will give it a shot 😂

@vody-am
Copy link
Contributor Author

vody-am commented Apr 30, 2024

@lvhan028 while I am here -- is there anything special one has to set to use multiple GPUs? Thus far I have experimented with one container per GPU, but it seems like there should be a way to use multiple cards. Is it the --tp setting?

@lvhan028
Copy link
Collaborator

lvhan028 commented May 1, 2024

Yes, by setting tp parameter

@lvhan028
Copy link
Collaborator

lvhan028 commented May 2, 2024

Hi, @vody-am
there are linting errors, which are able to fixed by running the following commands:

pip install pre-commit
cd the/root/path/of/lmdeploy
pre-commit install .
pre-commit run --all-files

@lvhan028 lvhan028 changed the title #1514 Update docker docs for VL api Update docker docs for VL api May 6, 2024
@lvhan028 lvhan028 merged commit 3a859fd into InternLM:main May 6, 2024
3 checks passed

COPY . .

CMD ["lmdeploy", "serve", "api_server", "liuhaotian/llava-v1.6-34b"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a real production environment, the deployed model path is usually stored on something like Open Storage Service. Almost never download and load from HuggingFace every time you deploy.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Therefore, this is more of a simple reference and has little significance in actual production environments.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's a fair assessment.

@vody-am vody-am deleted the docs/vlm_serving branch May 6, 2024 03:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants