[Refactor] Support multi-session chat #178

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

lvhan028 merged 28 commits into InternLM:main from wangruohui:multi_session

Aug 7, 2023

Collaborator

wangruohui commented Jul 28, 2023

No description provided.

wangruohui marked this pull request as ready for review

July 28, 2023 07:56

lvhan028 requested a review from AllentDan

July 31, 2023 09:50

lvhan028 reviewed

View reviewed changes

lmdeploy/pytorch/chat.py Show resolved Hide resolved

lvhan028 reviewed

View reviewed changes

lmdeploy/pytorch/model.py Outdated Show resolved Hide resolved

lvhan028 reviewed

View reviewed changes

lmdeploy/pytorch/model.py Outdated Show resolved Hide resolved

Collaborator

lvhan028 commented Aug 1, 2023

Can we put unittest in tests folder?

lvhan028 reviewed

View reviewed changes

lmdeploy/pytorch/dist.py Outdated Show resolved Hide resolved

lvhan028 reviewed

View reviewed changes

lmdeploy/pytorch/chat.py Show resolved Hide resolved

lvhan028 reviewed

View reviewed changes

lmdeploy/pytorch/adapters/__init__.py Show resolved Hide resolved

lvhan028 reviewed

View reviewed changes

lmdeploy/pytorch/adapters/llama2.py Outdated Show resolved Hide resolved

lvhan028 reviewed

View reviewed changes

lmdeploy/pytorch/adapters/llama2.py Show resolved Hide resolved

lvhan028 reviewed

View reviewed changes

lmdeploy/pytorch/chat.py Show resolved Hide resolved

lvhan028 reviewed

View reviewed changes

lmdeploy/pytorch/model.py Outdated Show resolved Hide resolved

Collaborator

lvhan028 commented Aug 1, 2023 •

edited

Loading

I tried python -m lmdeploy.pytorch.chat /nvme/shared_data/InternLM/7B/0705/5299 --accel deepspeed, but got errors like

TypeError: Subscripted generics cannot be used with class and instance checks

deepspeed --module --num_gpus 2 lmdeploy.pytorch.chat /nvme/shared_data/InternLM/7B/0705/5299 --accel deepspeed suffer from the same error, too

lvhan028 reviewed

View reviewed changes

lmdeploy/pytorch/session.py Show resolved Hide resolved

lvhan028 reviewed

View reviewed changes

lmdeploy/pytorch/session.py Outdated Show resolved Hide resolved

lvhan028 reviewed

View reviewed changes

lmdeploy/pytorch/utils.py Outdated Show resolved Hide resolved

lvhan028 added the enhancement label

Collaborator Author

wangruohui commented Aug 1, 2023

I tried python -m lmdeploy.pytorch.chat /nvme/shared_data/InternLM/7B/0705/5299 --accel deepspeed, but got errors like
TypeError: Subscripted generics cannot be used with class and instance checks
deepspeed --module --num_gpus 2 lmdeploy.pytorch.chat /nvme/shared_data/InternLM/7B/0705/5299 --accel deepspeed suffer from the same error, too

Solved

AllentDan reviewed

View reviewed changes

lmdeploy/pytorch/chat.py Outdated Show resolved Hide resolved

lmdeploy/pytorch/chat.py Show resolved Hide resolved

lmdeploy/pytorch/model.py Outdated Show resolved Hide resolved

lmdeploy/pytorch/model.py Outdated Show resolved Hide resolved

tests/pytorch/test_dist.py Outdated Show resolved Hide resolved

wangruohui mentioned this pull request

[Feature] Support decode with DP in pytorch #193

Merged

wangruohui added 9 commits

August 3, 2023 14:32


          add some dist utils

9a53c44


          add model utils

a3146f3


          add termio and basicstreamer

66ed2f8


          typo

a10f2ed


          fix world size

248f7b4


          refactor chat and tested llama1

5417e96


          add internlm adapter and support stoping criteria

0322f40


          concat with id for internlm

a272ed7


          update docstring

15a6025

wangruohui added 17 commits

August 3, 2023 14:32


          update and support llama2

1c9adc9


          typo

117b0ee


          move docs to docs

f59d565


          update docstring of session manager

8dfb221


          update docstring

b384b26


          update docs

a7add95


          fix accel none in model

097fc38


          fix and add test for tensor broadcast

6492fdc


          fix session using typing to check type


          add docstrings and comprehensive condition test

50b2571


          unit test for dist

b22ef94


          fix session

d76499e


          split unittests of utils

879659b


          typo

5d7a261


          update control flow of accel

c8fbc0b


          move test model

9b451e3


          remove main in unittest

d3c3fe4

wangruohui force-pushed the multi_session branch from 5c81542 to d3c3fe4 Compare

August 3, 2023 06:33


          remove some log

ba0f771

Collaborator Author

wangruohui commented Aug 3, 2023

rebased #188

lvhan028 approved these changes

View reviewed changes

AllentDan reviewed

View reviewed changes

tests/pytorch/test_model.py Outdated Show resolved Hide resolved

lmdeploy/pytorch/utils.py

Comment on lines +9 to +12

    
              try:

                  import readline  # To support command line history # noqa: F401

              except ImportError:  # readline not available

                  pass

Collaborator

AllentDan Aug 7, 2023

readline was not in use

Collaborator Author

wangruohui Aug 7, 2023

This is as expected because import readline just takes effect. After importing this module, user can retrieve history input via arrow keys, during chat. So we don't need to really use it as normal python packages, but need to import it.

lmdeploy/pytorch/model.py Show resolved Hide resolved

lmdeploy/pytorch/model.py Show resolved Hide resolved


          remove some comments

e5bbda6

AllentDan approved these changes

View reviewed changes

lvhan028 merged commit 4bd0b48 into InternLM:main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels