-
Notifications
You must be signed in to change notification settings - Fork 2.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Trainer] remove redundant memory metrics and set enable as default #8374
base: develop
Are you sure you want to change the base?
Conversation
Thanks for your contribution! |
paddlenlp/trainer/trainer.py
Outdated
logs["current_memory_allocated"] = current_memory_allocated / divisor | ||
logs["current_memory_reserved"] = current_memory_reserved / divisor | ||
logs["max_memory_allocated"] = max_memory_allocated / divisor | ||
logs["max_memory_reserved"] = max_memory_reserved / divisor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PaddleNLP/scripts/distribute/ci_case_dy.sh
Line 454 in 09a0ce7
mem=`cat $log_dir/workerlog.0 | grep 'global_step: 30' | awk -F 'gpu_max_memory_reserved: ' '{print $2}' | awk -F ',' '{print $1}'` |
这处也对应改了吧,gpu_max_memory_reserved -> max_memory_reserved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
已修改
paddlenlp/trainer/trainer.py
Outdated
max_memory_reserved = core.device_memory_stat_peak_value("Reserved", device_id) | ||
logs["current_memory_allocated"] = current_memory_allocated / divisor | ||
logs["current_memory_reserved"] = current_memory_reserved / divisor | ||
logs["max_memory_allocated"] = max_memory_allocated / divisor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
logs["max_memory_allocated"] = max_memory_allocated / divisor | |
logs["max_memory_allocated"] = max_memory_allocated >> 20 |
这个之前是MB单位,建议不要改变了,保持原来写法。用除法的话,是浮点数,还有小数位的问题,建议直接位运算,MB为单位
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
已修改
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #8374 +/- ##
===========================================
+ Coverage 55.36% 55.41% +0.04%
===========================================
Files 614 615 +1
Lines 96016 96241 +225
===========================================
+ Hits 53164 53335 +171
- Misses 42852 42906 +54 ☔ View full report in Codecov by Sentry. |
paddlenlp/trainer/training_args.py
Outdated
@@ -738,7 +738,7 @@ class TrainingArguments: | |||
metadata={"help": "The path to a folder with a valid checkpoint for your model."}, | |||
) | |||
skip_memory_metrics: bool = field( | |||
default=True, metadata={"help": "Whether or not to skip adding of memory profiler reports to metrics."} | |||
default=False, metadata={"help": "Whether or not to skip adding of memory profiler reports to metrics."} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这个别改了吧,你们需要用,自己打看
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR types
Others
PR changes
Others
Description
remove redundant memory metrics and enable memory metrics print as default