-
-
Notifications
You must be signed in to change notification settings - Fork 11.5k
[TPU] update TPU benchmark threshold #25713
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[TPU] update TPU benchmark threshold #25713
Conversation
9bec147 to
b19d058
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request updates the expected throughput threshold for a TPU benchmark. The change lowers the expected performance, which is a significant modification. My review focuses on improving the maintainability of this change by suggesting the addition of an inline comment to document the reason for this adjustment, preventing future confusion.
| MAX_MODEL_LEN=2048 | ||
| DOWNLOAD_DIR=/mnt/disks/persist | ||
| EXPECTED_THROUGHPUT=10.0 | ||
| EXPECTED_THROUGHPUT=8.9 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change lowers the expected throughput by over 10%, which is a significant performance regression. To ensure code maintainability and provide context for future developers, it's crucial to document the reason for this change directly in the configuration file. Adding a comment will clarify why the threshold was adjusted and prevent confusion or unnecessary investigations into this performance change in the future.
# Lowered from 10.0 to align with performance on the stable torch_xla version.
EXPECTED_THROUGHPUT=8.9
Signed-off-by: Johnny Yang <johnnyyang@google.com>
b19d058 to
bf98a7c
Compare
yaochengji
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
Signed-off-by: Johnny Yang <johnnyyang@google.com>
Signed-off-by: Johnny Yang <johnnyyang@google.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
Signed-off-by: Johnny Yang <johnnyyang@google.com> Signed-off-by: Dhruvil Bhatt <bhattdbh@amazon.com>
Signed-off-by: Johnny Yang <johnnyyang@google.com>
Signed-off-by: Johnny Yang <johnnyyang@google.com>
Signed-off-by: Johnny Yang <johnnyyang@google.com> Signed-off-by: xuebwang-amd <xuebwang@amd.com>
Signed-off-by: Johnny Yang <johnnyyang@google.com>
Purpose
Update TPU benchmark expected threshold for stable version of torch_xla
Test Result
CI
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.