Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feat] Support update weights without restart server #1157

Merged
merged 12 commits into from
Aug 20, 2024

Conversation

shanyu-sys
Copy link
Collaborator

@shanyu-sys shanyu-sys commented Aug 20, 2024

Motivation

Support updating the model weights without relaunching the server.
#634

Modification

  • Support post a "/update_weights" request in server.py
  • Add update_weights methods in tokenizer_manager, tp_worker and model_runner
  • Add unit test in test_update_weights.py

Checklist

  • Before submitting a PR for review, make sure it has passed verification in your local development environment at least.
  • Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  • Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  • Modify documentation as needed, such as docstrings or example tutorials.

@shanyu-sys shanyu-sys mentioned this pull request Aug 20, 2024
29 tasks
@Ying1123 Ying1123 self-assigned this Aug 20, 2024
test/srt/test_update_weights.py Show resolved Hide resolved
@Ying1123 Ying1123 merged commit cd10654 into sgl-project:main Aug 20, 2024
5 checks passed
@Ying1123 Ying1123 deleted the update_weights branch August 20, 2024 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants