[Sync Pipeline Inference] Sync pipeline inference branch to main #4820

FoolPlayer · 2023-09-27T08:32:53Z

📌 Checklist before creating the PR

I have created an issue for this PR for traceability
The title follows the standard format: [doc/gemini/tensor/...]: A concise description
I have added relevant tags if possible for us to better distinguish different PRs

🚨 Issue number

Link this PR to your issue with words like fixed to automatically close the linked issue upon merge

e.g. fixed #1234, closed #1234, resolved #1234

📝 What does this PR do?

Summarize your work here.
if you have any plots/diagrams/screenshots/tables, please attach them here.

💥 Checklist before requesting a review

I have linked my PR to an issue (instruction)
My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
I have performed a self-review of my code
I have added thorough tests.
I have added docstrings for all the functions/methods I implemented

⭐️ Do you enjoy contributing to Colossal-AI?

🌝 Yes, I do.
🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

* add pp stage manager as circle stage * fix a bug when create process group * add ppinfer basic framework * add micro batch manager and support kvcache-pp gpt2 fwd * add generate schedule * use mb size to control mb number * support generate with kv cache * add output, remove unused code * add test * reuse shardformer to build model * refactor some code and use the same attribute name of hf * fix review and add test for generation * remove unused file * fix CI * add cache clear * fix code error * fix typo

* add pp stage manager as circle stage * fix a bug when create process group * add ppinfer basic framework * add micro batch manager and support kvcache-pp gpt2 fwd * add generate schedule * use mb size to control mb number * support generate with kv cache * add output, remove unused code * add test * reuse shardformer to build model * refactor some code and use the same attribute name of hf * fix review and add test for generation * remove unused file * modify the way of saving newtokens * modify to tieweight * modify test * remove unused file * solve review * add docstring

* support llama pipeline inference * remove tie weight operation

… 2 (#4708) * add benchmark verbose * fix export tokens * fix benchmark verbose * add P2POp style to do p2p communication * modify schedule as p2p type when ppsize is 2 * remove unused code and add docstring

* add benchmark script * update argparse * fix fp16 load * refactor code style * add docstring * polish code * fix test bug

* add readme doc * add a ico * Add performance * update table of contents

github-actions · 2023-09-27T11:11:03Z

The code coverage for the changed files is 68%.

Click me to view the complete report

Name                                                            Stmts   Miss  Cover
-----------------------------------------------------------------------------------
colossalai/inference/__init__.py                                    2      0   100%
colossalai/inference/pipeline/__init__.py                           2      0   100%
colossalai/inference/pipeline/engine.py                            34      3    91%
colossalai/inference/pipeline/microbatch_manager.py               117      4    97%
colossalai/inference/pipeline/modeling/__init__.py                  0      0   100%
colossalai/inference/pipeline/modeling/gpt2.py                    124     43    65%
colossalai/inference/pipeline/modeling/llama.py                    91     91     0%
colossalai/inference/pipeline/policy/gpt2_ppinfer.py               43      5    88%
colossalai/inference/pipeline/utils.py                             15     15     0%
colossalai/pipeline/p2p.py                                        137     46    66%
colossalai/pipeline/schedule/generate.py                          191     84    56%
colossalai/pipeline/stage_manager.py                               49      0   100%
tests/test_checkpoint_io/test_low_level_zero_checkpoint_io.py      59      1    98%
tests/test_generate/test_pipeline_infer.py                         43      1    98%
-----------------------------------------------------------------------------------
TOTAL                                                             907    293    68%

colossalai/pipeline/schedule/generate.py

tests/test_generate/test_pipeline_infer.py

github-actions · 2023-10-09T10:31:43Z

The code coverage for the changed files is 68%.

Click me to view the complete report

Name                                                            Stmts   Miss  Cover
-----------------------------------------------------------------------------------
colossalai/inference/__init__.py                                    2      0   100%
colossalai/inference/pipeline/__init__.py                           2      0   100%
colossalai/inference/pipeline/engine.py                            34      3    91%
colossalai/inference/pipeline/microbatch_manager.py               117      4    97%
colossalai/inference/pipeline/modeling/__init__.py                  0      0   100%
colossalai/inference/pipeline/modeling/gpt2.py                    124     43    65%
colossalai/inference/pipeline/modeling/llama.py                    91     91     0%
colossalai/inference/pipeline/policy/gpt2_ppinfer.py               43      5    88%
colossalai/inference/pipeline/utils.py                             15     15     0%
colossalai/pipeline/p2p.py                                        137     46    66%
colossalai/pipeline/schedule/generate.py                          191     84    56%
colossalai/pipeline/stage_manager.py                               49      0   100%
tests/test_checkpoint_io/test_low_level_zero_checkpoint_io.py      59      1    98%
tests/test_infer/test_pipeline_infer.py                            43      1    98%
-----------------------------------------------------------------------------------
TOTAL                                                             907    293    68%

…h#4820) * [pipeline inference] pipeline inference (hpcaitech#4492) * add pp stage manager as circle stage * fix a bug when create process group * add ppinfer basic framework * add micro batch manager and support kvcache-pp gpt2 fwd * add generate schedule * use mb size to control mb number * support generate with kv cache * add output, remove unused code * add test * reuse shardformer to build model * refactor some code and use the same attribute name of hf * fix review and add test for generation * remove unused file * fix CI * add cache clear * fix code error * fix typo * [Pipeline inference] Modify to tieweight (hpcaitech#4599) * add pp stage manager as circle stage * fix a bug when create process group * add ppinfer basic framework * add micro batch manager and support kvcache-pp gpt2 fwd * add generate schedule * use mb size to control mb number * support generate with kv cache * add output, remove unused code * add test * reuse shardformer to build model * refactor some code and use the same attribute name of hf * fix review and add test for generation * remove unused file * modify the way of saving newtokens * modify to tieweight * modify test * remove unused file * solve review * add docstring * [Pipeline inference] support llama pipeline inference (hpcaitech#4647) * support llama pipeline inference * remove tie weight operation * [pipeline inference] Fix the blocking of communication when ppsize is 2 (hpcaitech#4708) * add benchmark verbose * fix export tokens * fix benchmark verbose * add P2POp style to do p2p communication * modify schedule as p2p type when ppsize is 2 * remove unused code and add docstring * [Pipeline inference] Refactor code, add docsting, fix bug (hpcaitech#4790) * add benchmark script * update argparse * fix fp16 load * refactor code style * add docstring * polish code * fix test bug * [Pipeline inference] Add pipeline inference docs (hpcaitech#4817) * add readme doc * add a ico * Add performance * update table of contents * refactor code (hpcaitech#4873)

FoolPlayer added 6 commits September 27, 2023 17:36

[Pipeline inference] support llama pipeline inference (#4647)

84e76c1

* support llama pipeline inference * remove tie weight operation

[pipeline inference] Fix the blocking of communication when ppsize is…

65d300f

… 2 (#4708) * add benchmark verbose * fix export tokens * fix benchmark verbose * add P2POp style to do p2p communication * modify schedule as p2p type when ppsize is 2 * remove unused code and add docstring

[Pipeline inference] Refactor code, add docsting, fix bug (#4790)

121ab52

* add benchmark script * update argparse * fix fp16 load * refactor code style * add docstring * polish code * fix test bug

[Pipeline inference] Add pipeline inference docs (#4817)

9418c07

* add readme doc * add a ico * Add performance * update table of contents

FoolPlayer force-pushed the feature/pipeline-infer branch from dbfa6e1 to 9418c07 Compare September 27, 2023 09:38

ver217 reviewed Oct 9, 2023

View reviewed changes

colossalai/pipeline/schedule/generate.py Outdated Show resolved Hide resolved

ver217 reviewed Oct 9, 2023

View reviewed changes

tests/test_generate/test_pipeline_infer.py Outdated Show resolved Hide resolved

refactor code (#4873)

139adda

ver217 approved these changes Oct 9, 2023

View reviewed changes

Orion-Zheng requested review from CjhHa1 and tiandiao123 October 9, 2023 08:08

CjhHa1 approved these changes Oct 10, 2023

View reviewed changes

FoolPlayer merged commit 08a9f76 into main Oct 11, 2023
6 of 7 checks passed

ver217 deleted the feature/pipeline-infer branch October 13, 2023 09:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Sync Pipeline Inference] Sync pipeline inference branch to main #4820

[Sync Pipeline Inference] Sync pipeline inference branch to main #4820

FoolPlayer commented Sep 27, 2023 •

edited

Loading

github-actions bot commented Sep 27, 2023

github-actions bot commented Oct 9, 2023

[Sync Pipeline Inference] Sync pipeline inference branch to main #4820

[Sync Pipeline Inference] Sync pipeline inference branch to main #4820

Conversation

FoolPlayer commented Sep 27, 2023 • edited Loading

📌 Checklist before creating the PR

🚨 Issue number

📝 What does this PR do?

💥 Checklist before requesting a review

⭐️ Do you enjoy contributing to Colossal-AI?

github-actions bot commented Sep 27, 2023

github-actions bot commented Oct 9, 2023

FoolPlayer commented Sep 27, 2023 •

edited

Loading