[AutoParallel] Optimize Reshard [part2: optimize _reshard_input] #60022
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
你的PR提交成功，感谢你对开源项目的贡献! |
AndSonder
changed the title
AndSonder
changed the title
From00
reviewed
Dec 15, 2023
| @@ -341,7 +341,9 @@ def insert_cast_op(block, idx, tensor, op_role, tensor_type): | |||
| type=tensor.type, | |||
| lod_level=tensor.lod_level, | |||
| ) | |||
| cast_op = block._insert_op( | |||
|
|
|||
| insert_op = block._insert_op if sync else block._insert_op_without_sync | |||
There was a problem hiding this comment.
这里insert_op和cast_op使用相同的命名结构，但却完全是两个不同的语义。前者是一个函数，后者是一个OP对象，放在一起很容易让人混淆。建议做更清晰的区分。
There was a problem hiding this comment.
将 insert_op 改成了 insert_operation。cast_op 里面的 op 表示 operator，可以区分二者的区别
HermitSun
pushed a commit
to HermitSun/Paddle
that referenced
this pull request
Dec 21, 2023
…dlePaddle#60022) * opt reshard parse_op_desc * recover third party * change var name * fix code style
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Performance optimization
PR changes
Others
Description
静态图模式下分析得，_reshard_input 中的 parse_op_desc 里面调用了很多 sync_with_cpp 消耗了较多的时间，耗时图如下所示：
经过分析可得，parse_op_desc 中 Insert 类的操作 如insert_c_concat_op、insert_slice_op、insert_concat_op等操作，每次插入op之前都会和cpp端进行同步，这是没有必要的，我们只需要在op插入完成之后做一次同步即可。经模型验证，该优化不会影响模型精度，模型精度可与优化前对齐。
本地测试优化后模型实际Run之前的耗时，优化幅度达 0.208 倍 （23.889 -> 18.928）。
测试环境：本地四卡 1080Ti机器，PaddleNLP Llama2 7b模型, 静态图模式（hack config: config.num_hidden_layers = 12）
优化前后 cprofiler 结果如下,
<-- !!!指向的是被优化的项：优化后的耗时可视化如下：
脚本启动命令：
python -u -m paddle.distributed.launch \ --gpus "0,1,2,3" \ --log_dir "output/$task_name""_log" \ auto_parallel/run_pretrain_auto.py \ --model_type "llama" \ --model_name_or_path "facebook/llama-7b" \ --tokenizer_name_or_path "facebook/llama-7b" \ --input_dir "./data" \ --output_dir "output/$task_name" \ --split 949,50,1 \ --max_seq_length 2048 \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 4 \ --gradient_accumulation_steps 16 \ --fp16 1 \ --fp16_opt_level "O2" \ --scale_loss 1024 \ --pipeline_parallel_degree 2 \ --tensor_parallel_degree 2 \ --sharding_parallel_degree 1 \ --sharding "stage1" \ --learning_rate 0.0001 \ --min_learning_rate 0.00001 \ --max_steps 30 \ --save_steps 5000 \ --weight_decay 0.01 \ --warmup_ratio 0.01 \ --max_grad_norm 1.0 \ --logging_steps 1\ --dataloader_num_workers 1 \ --sharding "" \ --eval_steps 1000 \ --report_to "visualdl" \ --disable_tqdm true \ --continue_training 0\ --recompute 1 \ --do_train 1 \ --do_eval 0 \ --device "gpu" \ --data_impl "mmap" \ --parallel_mode "auto" \ --fuse_attention_qkv 1 \ --fuse_attention_ffn 1 \ --use_fused_rope 1 \ --use_flash_attention 1