[Feature] Support Sequence parallel #456
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pppppM
changed the title
pppppM
approved these changes
Mar 22, 2024
pppppM
pushed a commit
to pppppM/xtuner
that referenced
this pull request
Mar 29, 2024
* support sequence
* add configs
* add sp example to custom dataset
* WIP
* add dispatch utils
* delete useless codes
* move xtuner/engine/sequence_parallel to xtuner/parallel/sequence
* fix lint
* fix lint
* add init_dist to xtuner and add trust_remote_code=True to AutoConfig
* add internlm2 custom_dataset sp4 config
* Sequence Parallel doc V1
* Sequence Parallel doc V1
* Sequence Parallel doc V1
* fix bugs in llama_varlen_attn_forward
* rename indexes to position_ids
* add attn_implementation to config
* delete useless codes
* fix lint
* refine default_collate_fn
* refine doc
* refine doc
* refine doc
* delete replace_internlm2_rote
* add repeat_kv_bshd
* fix apply_rotary_pos_emb bug
* add enable_sequence_parallel flag
* refine doc
* assert {'input_ids', 'labels'}.issubset(dataset.column_names)
* refine doc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
【已完成】
a. 分布式环境初始化
b. 数据pad与切分
c. data sampler 适配
d. Attention 部分适配序列并行
e. 训练loss sequence parallel group 内reduce
【待完成】
完善文档