Add request distributor server #903
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tpoisonooo
approved these changes
Jan 8, 2024
lvhan028
reviewed
Jan 11, 2024
|
Run command: python3 lmdeploy/serve/proxy/proxy.py --server-port 33338got error message: |
|
miss |
lvhan028
reviewed
Jan 11, 2024
lmdeploy/constants.py
Outdated
| @@ -0,0 +1,39 @@ | |||
| # Copyright (c) OpenMMLab. All rights reserved. | |||
There was a problem hiding this comment.
Is it better to put constants.py to lmdeploy/serve/proxy?
Conflicts: docs/en/serving/restful_api.md docs/zh_cn/serving/restful_api.md
lvhan028
reviewed
Jan 11, 2024
docs/en/serving/proxy_server.md
Outdated
| @@ -0,0 +1,39 @@ | |||
| ## Proxy | |||
There was a problem hiding this comment.
H1 title: "Request Distributor Server"
lvhan028
reviewed
Jan 11, 2024
docs/en/serving/restful_api.md
Outdated
| @@ -162,3 +162,7 @@ lmdeploy serve gradio api_server_url --server_name ${gradio_ui_ip} --server_port | |||
| 5. If you need to adjust other default parameters of the session, such as the content of fields like system. You can directly pass in the initialization parameters of the [dialogue template](https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/model.py). For example, for the internlm-chat-7b model, you can set the `--meta_instruction` parameter when starting the `api_server`. | |||
|
|
|||
| 6. Regarding the stop words, we only support characters that encode into a single index. Furthermore, there may be multiple indexes that decode into results containing the stop word. In such cases, if the number of these indexes is too large, we will only use the index encoded by the tokenizer. If you want use a stop symbol that encodes into multiple indexes, you may consider performing string matching on the streaming client side. Once a successful match is found, you can then break out of the streaming loop. | |||
|
|
|||
| ### multiple services | |||
There was a problem hiding this comment.
request distribution service
lvhan028
reviewed
Jan 11, 2024
docs/zh_cn/serving/restful_api.md
Outdated
| @@ -156,3 +156,7 @@ lmdeploy serve gradio api_server_url --server_name ${gradio_ui_ip} --server_port | |||
| 5. 如需调整会话默认的其他参数，比如 system 等字段的内容，可以直接将[对话模板](https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/model.py)初始化参数传入。比如 internlm-chat-7b 模型，可以通过启动`api_server`时，设置`--meta_instruction`参数。 | |||
|
|
|||
| 6. 关于停止符，我们只支持编码后为单个 index 的字符。此外，可能存在多种 index 都会解码出带有停止符的结果。对于这种情况，如果这些 index 数量太多，我们只会采用 tokenizer 编码出的 index。而如果你想要编码后为多个 index 的停止符，可以考虑在流式客户端做字符串匹配，匹配成功后跳出流式循环即可。 | |||
|
|
|||
| ### 多个服务并行 | |||
lvhan028
reviewed
Jan 11, 2024
docs/en/serving/proxy_server.md
Outdated
| Start the proxy service: | ||
|
|
||
| ```shell | ||
| python lmdeploy/serve/proxy/proxy.py --server_name {server_name} --server_port {server_port} --strategy "min_expected_latency" |
There was a problem hiding this comment.
能用下面这种方式么？
python3 -m lmdeploy.serve.proxy --server-name {server_name} --server-port
lvhan028
approved these changes
Jan 12, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.