Skip to content

[Inductor] Performance smoke test - hf bert performance increased #94088

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Parent: Add extra debug info for the super common "index out of bounds" error.
wants to merge 1 commit into from
Closed

[Inductor] Performance smoke test - hf bert performance increased #94088

wants to merge 1 commit into from

Conversation

@weiwangmeta
Copy link
Contributor

@weiwangmeta weiwangmeta commented Feb 3, 2023 •
edited
Loading

therefore bumping up from 1.185 to 1.200 to better detect regression
logurl date model speedup
https://ossci-raw-job-status.s3.amazonaws.com/log/11101705328 2023-02-03T23:05:19.5738026Z hf_Bert 1.2122
https://ossci-raw-job-status.s3.amazonaws.com/log/11101331469 2023-02-03T22:54:18.0252738Z hf_Bert 1.2129
https://ossci-raw-job-status.s3.amazonaws.com/log/11101288841 2023-02-03T22:52:17.6331332Z hf_Bert 1.2189
https://ossci-raw-job-status.s3.amazonaws.com/log/11101190372 2023-02-03T22:50:28.6010460Z hf_Bert 1.2117
https://ossci-raw-job-status.s3.amazonaws.com/log/11101101525 2023-02-03T22:27:18.5573576Z hf_Bert 1.2088
https://ossci-raw-job-status.s3.amazonaws.com/log/11101034545 2023-02-03T22:24:33.8710157Z hf_Bert 1.2229
https://ossci-raw-job-status.s3.amazonaws.com/log/11101004878 2023-02-03T22:22:38.0506379Z hf_Bert 1.2074
https://ossci-raw-job-status.s3.amazonaws.com/log/11100834787 2023-02-03T22:12:34.9376779Z hf_Bert 1.2142
https://ossci-raw-job-status.s3.amazonaws.com/log/11100413479 2023-02-03T21:47:55.7536822Z hf_Bert 1.2112
https://ossci-raw-job-status.s3.amazonaws.com/log/11100372087 2023-02-03T21:46:19.6411599Z hf_Bert 1.2175
https://ossci-raw-job-status.s3.amazonaws.com/log/11100291417 2023-02-03T21:41:01.3427726Z hf_Bert 1.2068
https://ossci-raw-job-status.s3.amazonaws.com/log/11100137256 2023-02-03T21:32:14.4491714Z hf_Bert 1.2089
https://ossci-raw-job-status.s3.amazonaws.com/log/11098980986 2023-02-03T20:30:13.4082966Z hf_Bert 1.2109
https://ossci-raw-job-status.s3.amazonaws.com/log/11098634747 2023-02-03T20:12:57.4921305Z hf_Bert 1.2169
https://ossci-raw-job-status.s3.amazonaws.com/log/11096295932 2023-02-03T18:58:55.1214750Z hf_Bert 1.2196
https://ossci-raw-job-status.s3.amazonaws.com/log/11095904757 2023-02-03T18:49:48.4541355Z hf_Bert 1.22
https://ossci-raw-job-status.s3.amazonaws.com/log/11095292402 2023-02-03T18:10:54.6924201Z hf_Bert 1.2122
https://ossci-raw-job-status.s3.amazonaws.com/log/11095026691 2023-02-03T18:11:26.7384107Z hf_Bert 1.2228
https://ossci-raw-job-status.s3.amazonaws.com/log/11094943489 2023-02-03T17:53:00.0989341Z hf_Bert 1.2165
https://ossci-raw-job-status.s3.amazonaws.com/log/11093227145 2023-02-03T16:04:18.7935799Z hf_Bert 1.2208
https://ossci-raw-job-status.s3.amazonaws.com/log/11092910912 2023-02-03T15:51:28.1977577Z hf_Bert 1.2188
https://ossci-raw-job-status.s3.amazonaws.com/log/11091775528 2023-02-03T15:27:21.7984395Z hf_Bert 1.2231
https://ossci-raw-job-status.s3.amazonaws.com/log/11091768252 2023-02-03T15:12:33.0339859Z hf_Bert 1.2167
https://ossci-raw-job-status.s3.amazonaws.com/log/11091051563 2023-02-03T14:44:42.7011287Z hf_Bert 1.2214
https://ossci-raw-job-status.s3.amazonaws.com/log/11088539227 2023-02-03T12:41:29.9098435Z hf_Bert 1.2192
https://ossci-raw-job-status.s3.amazonaws.com/log/11088428613 2023-02-03T12:35:38.4674850Z hf_Bert 1.2108
https://ossci-raw-job-status.s3.amazonaws.com/log/11088405279 2023-02-03T12:34:54.0870617Z hf_Bert 1.2197
https://ossci-raw-job-status.s3.amazonaws.com/log/11087037337 2023-02-03T12:06:58.2426787Z hf_Bert 1.2174
https://ossci-raw-job-status.s3.amazonaws.com/log/11085381881 2023-02-03T10:19:20.8764019Z hf_Bert 1.2189
https://ossci-raw-job-status.s3.amazonaws.com/log/11085190037 2023-02-03T10:14:41.5234245Z hf_Bert 1.2046
https://ossci-raw-job-status.s3.amazonaws.com/log/11085016390 2023-02-03T09:50:59.7484273Z hf_Bert 1.2155
https://ossci-raw-job-status.s3.amazonaws.com/log/11084948754 2023-02-03T09:47:15.7358069Z hf_Bert 1.2083
https://ossci-raw-job-status.s3.amazonaws.com/log/11084675155 2023-02-03T09:42:35.6628268Z hf_Bert 1.2126
https://ossci-raw-job-status.s3.amazonaws.com/log/11081270865 2023-02-03T06:05:22.1828269Z hf_Bert 1.2083
https://ossci-raw-job-status.s3.amazonaws.com/log/11081252914 2023-02-03T05:43:59.0680872Z hf_Bert 1.2097
https://ossci-raw-job-status.s3.amazonaws.com/log/11081252670 2023-02-03T05:44:17.0945428Z hf_Bert 1.2143
https://ossci-raw-job-status.s3.amazonaws.com/log/11081244430 2023-02-03T05:43:43.6811750Z hf_Bert 1.2204
https://ossci-raw-job-status.s3.amazonaws.com/log/11081191493 2023-02-03T05:38:43.7833293Z hf_Bert 1.2079
https://ossci-raw-job-status.s3.amazonaws.com/log/11081191168 2023-02-03T05:38:21.1397044Z hf_Bert 1.2067
https://ossci-raw-job-status.s3.amazonaws.com/log/11081189846 2023-02-03T05:38:53.5914557Z hf_Bert 1.2073
https://ossci-raw-job-status.s3.amazonaws.com/log/11080883297 2023-02-03T05:13:25.0077772Z hf_Bert 1.2105
https://ossci-raw-job-status.s3.amazonaws.com/log/11080456108 2023-02-03T04:34:34.0934838Z hf_Bert 1.204
https://ossci-raw-job-status.s3.amazonaws.com/log/11079957300 2023-02-03T03:53:18.9091026Z hf_Bert 1.207
https://ossci-raw-job-status.s3.amazonaws.com/log/11078579407 2023-02-03T02:03:11.2254812Z hf_Bert 1.2049
https://ossci-raw-job-status.s3.amazonaws.com/log/11078204621 2023-02-03T01:58:39.0887941Z hf_Bert 1.2214
https://ossci-raw-job-status.s3.amazonaws.com/log/11078126527 2023-02-03T01:38:20.2183225Z hf_Bert 1.2061
https://ossci-raw-job-status.s3.amazonaws.com/log/11077409013 2023-02-03T00:48:51.8981496Z hf_Bert 1.2086
https://ossci-raw-job-status.s3.amazonaws.com/log/11077176061 2023-02-03T00:27:27.2594172Z hf_Bert 1.2077
https://ossci-raw-job-status.s3.amazonaws.com/log/11077075809 2023-02-03T00:21:54.4916449Z hf_Bert 1.2103
https://ossci-raw-job-status.s3.amazonaws.com/log/11076629886 2023-02-02T23:50:38.3512367Z hf_Bert 1.2191
https://ossci-raw-job-status.s3.amazonaws.com/log/11076577074 2023-02-02T23:46:06.5987589Z hf_Bert 1.2061
https://ossci-raw-job-status.s3.amazonaws.com/log/11076403972 2023-02-02T23:35:49.7931367Z hf_Bert 1.2088
https://ossci-raw-job-status.s3.amazonaws.com/log/11076234469 2023-02-02T23:25:55.7300688Z hf_Bert 1.2099
https://ossci-raw-job-status.s3.amazonaws.com/log/11075752070 2023-02-02T22:57:25.4280216Z hf_Bert 1.2048
https://ossci-raw-job-status.s3.amazonaws.com/log/11074434992 2023-02-02T22:10:58.4127805Z hf_Bert 1.2084
https://ossci-raw-job-status.s3.amazonaws.com/log/11074370082 2023-02-02T22:10:06.8153498Z hf_Bert 1.2075
https://ossci-raw-job-status.s3.amazonaws.com/log/11073914614 2023-02-02T21:25:53.3262334Z hf_Bert 1.2058
https://ossci-raw-job-status.s3.amazonaws.com/log/11073616418 2023-02-02T21:12:03.0024412Z hf_Bert 1.2053
https://ossci-raw-job-status.s3.amazonaws.com/log/11072632121 2023-02-02T20:25:37.5689220Z hf_Bert 1.2082
https://ossci-raw-job-status.s3.amazonaws.com/log/11072091471 2023-02-02T20:00:08.5175281Z hf_Bert 1.2079
https://ossci-raw-job-status.s3.amazonaws.com/log/11069395867 2023-02-02T18:29:04.6481423Z hf_Bert 1.2071
https://ossci-raw-job-status.s3.amazonaws.com/log/11069169921 2023-02-02T18:18:36.5701242Z hf_Bert 1.2036
https://ossci-raw-job-status.s3.amazonaws.com/log/11069070631 2023-02-02T18:15:32.2345859Z hf_Bert 1.2055
https://ossci-raw-job-status.s3.amazonaws.com/log/11067153829 2023-02-02T16:38:27.4201129Z hf_Bert 1.2133
https://ossci-raw-job-status.s3.amazonaws.com/log/11066885021 2023-02-02T16:28:44.4489971Z hf_Bert 1.2043

The above are the result of running a rockset query which returns links to the log and wget the logs and grep "Z hf_Bert"

cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire

@weiwangmeta
hf bert performance increased. below shows the last 65 runs from master
ce5d84e 
therefore bumping up from 1.185 to 1.200 to better detect regression
date                          model                             speedup
2023-02-03T23:05:19.5738026Z hf_Bert                            1.2122
2023-02-03T22:54:18.0252738Z hf_Bert                            1.2129
2023-02-03T22:52:17.6331332Z hf_Bert                            1.2189
2023-02-03T22:50:28.6010460Z hf_Bert                            1.2117
2023-02-03T22:27:18.5573576Z hf_Bert                            1.2088
2023-02-03T22:24:33.8710157Z hf_Bert                            1.2229
2023-02-03T22:22:38.0506379Z hf_Bert                            1.2074
2023-02-03T22:12:34.9376779Z hf_Bert                            1.2142
2023-02-03T21:47:55.7536822Z hf_Bert                            1.2112
2023-02-03T21:46:19.6411599Z hf_Bert                            1.2175
2023-02-03T21:41:01.3427726Z hf_Bert                            1.2068
2023-02-03T21:32:14.4491714Z hf_Bert                            1.2089
2023-02-03T20:30:13.4082966Z hf_Bert                            1.2109
2023-02-03T20:12:57.4921305Z hf_Bert                            1.2169
2023-02-03T18:58:55.1214750Z hf_Bert                            1.2196
2023-02-03T18:49:48.4541355Z hf_Bert                            1.22
2023-02-03T18:10:54.6924201Z hf_Bert                            1.2122
2023-02-03T18:11:26.7384107Z hf_Bert                            1.2228
2023-02-03T17:53:00.0989341Z hf_Bert                            1.2165
2023-02-03T16:04:18.7935799Z hf_Bert                            1.2208
2023-02-03T15:51:28.1977577Z hf_Bert                            1.2188
2023-02-03T15:27:21.7984395Z hf_Bert                            1.2231
2023-02-03T15:12:33.0339859Z hf_Bert                            1.2167
2023-02-03T14:44:42.7011287Z hf_Bert                            1.2214
2023-02-03T12:41:29.9098435Z hf_Bert                            1.2192
2023-02-03T12:35:38.4674850Z hf_Bert                            1.2108
2023-02-03T12:34:54.0870617Z hf_Bert                            1.2197
2023-02-03T12:06:58.2426787Z hf_Bert                            1.2174
2023-02-03T10:19:20.8764019Z hf_Bert                            1.2189
2023-02-03T10:14:41.5234245Z hf_Bert                            1.2046
2023-02-03T09:50:59.7484273Z hf_Bert                            1.2155
2023-02-03T09:47:15.7358069Z hf_Bert                            1.2083
2023-02-03T09:42:35.6628268Z hf_Bert                            1.2126
2023-02-03T06:05:22.1828269Z hf_Bert                            1.2083
2023-02-03T05:43:59.0680872Z hf_Bert                            1.2097
2023-02-03T05:44:17.0945428Z hf_Bert                            1.2143
2023-02-03T05:43:43.6811750Z hf_Bert                            1.2204
2023-02-03T05:38:43.7833293Z hf_Bert                            1.2079
2023-02-03T05:38:21.1397044Z hf_Bert                            1.2067
2023-02-03T05:38:53.5914557Z hf_Bert                            1.2073
2023-02-03T05:13:25.0077772Z hf_Bert                            1.2105
2023-02-03T04:34:34.0934838Z hf_Bert                            1.204
2023-02-03T03:53:18.9091026Z hf_Bert                            1.207
2023-02-03T02:03:11.2254812Z hf_Bert                            1.2049
2023-02-03T01:58:39.0887941Z hf_Bert                            1.2214
2023-02-03T01:38:20.2183225Z hf_Bert                            1.2061
2023-02-03T00:48:51.8981496Z hf_Bert                            1.2086
2023-02-03T00:27:27.2594172Z hf_Bert                            1.2077
2023-02-03T00:21:54.4916449Z hf_Bert                            1.2103
2023-02-02T23:50:38.3512367Z hf_Bert                            1.2191
2023-02-02T23:46:06.5987589Z hf_Bert                            1.2061
2023-02-02T23:35:49.7931367Z hf_Bert                            1.2088
2023-02-02T23:25:55.7300688Z hf_Bert                            1.2099
2023-02-02T22:57:25.4280216Z hf_Bert                            1.2048
2023-02-02T22:10:58.4127805Z hf_Bert                            1.2084
2023-02-02T22:10:06.8153498Z hf_Bert                            1.2075
2023-02-02T21:25:53.3262334Z hf_Bert                            1.2058
2023-02-02T21:12:03.0024412Z hf_Bert                            1.2053
2023-02-02T20:25:37.5689220Z hf_Bert                            1.2082
2023-02-02T20:00:08.5175281Z hf_Bert                            1.2079
2023-02-02T18:29:04.6481423Z hf_Bert                            1.2071
2023-02-02T18:18:36.5701242Z hf_Bert                            1.2036
2023-02-02T18:15:32.2345859Z hf_Bert                            1.2055
2023-02-02T16:38:27.4201129Z hf_Bert                            1.2133
2023-02-02T16:28:44.4489971Z hf_Bert                            1.2043
@weiwangmeta weiwangmeta requested review from a team, desertfire and ngimel February 3, 2023 23:30
@pytorch-bot
Copy link

pytorch-bot bot commented Feb 3, 2023 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/94088

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Failures

As of commit ce5d84e:

NEW FAILURES - The following jobs have failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@weiwangmeta weiwangmeta added the topic: not user facing topic category label Feb 3, 2023
@weiwangmeta
Copy link
Contributor Author

31 more runs:

https://ossci-raw-job-status.s3.amazonaws.com/log/11127831911 2023-02-06T06:20:56.8557970Z hf_Bert 1.2061
https://ossci-raw-job-status.s3.amazonaws.com/log/11122881503 2023-02-05T22:24:10.2077453Z hf_Bert 1.2083
https://ossci-raw-job-status.s3.amazonaws.com/log/11122506624 2023-02-05T21:29:47.4357319Z hf_Bert 1.2097
https://ossci-raw-job-status.s3.amazonaws.com/log/11117429569 2023-02-05T10:08:43.1206202Z hf_Bert 1.2112
https://ossci-raw-job-status.s3.amazonaws.com/log/11115977840 2023-02-05T06:30:12.8376346Z hf_Bert 1.2128
https://ossci-raw-job-status.s3.amazonaws.com/log/11115667574 2023-02-05T05:40:25.6277635Z hf_Bert 1.2214
https://ossci-raw-job-status.s3.amazonaws.com/log/11115649542 2023-02-05T05:37:11.4638343Z hf_Bert 1.2111
https://ossci-raw-job-status.s3.amazonaws.com/log/11115613263 2023-02-05T05:31:54.7687163Z hf_Bert 1.2092
https://ossci-raw-job-status.s3.amazonaws.com/log/11115337008 2023-02-05T04:48:21.2819979Z hf_Bert 1.218
https://ossci-raw-job-status.s3.amazonaws.com/log/11115140177 2023-02-05T04:21:25.9018320Z hf_Bert 1.212
https://ossci-raw-job-status.s3.amazonaws.com/log/11112485878 2023-02-04T22:01:22.0393540Z hf_Bert 1.2089
https://ossci-raw-job-status.s3.amazonaws.com/log/11112190187 2023-02-04T21:21:03.4568814Z hf_Bert 1.2107
https://ossci-raw-job-status.s3.amazonaws.com/log/11111996172 2023-02-04T20:50:49.7441057Z hf_Bert 1.2137
https://ossci-raw-job-status.s3.amazonaws.com/log/11111786921 2023-02-04T20:23:30.6567267Z hf_Bert 1.2061
https://ossci-raw-job-status.s3.amazonaws.com/log/11108926749 2023-02-04T13:42:38.3982822Z hf_Bert 1.2141
https://ossci-raw-job-status.s3.amazonaws.com/log/11108413666 2023-02-04T12:35:16.9019480Z hf_Bert 1.2084
https://ossci-raw-job-status.s3.amazonaws.com/log/11108223113 2023-02-04T12:10:45.9372374Z hf_Bert 1.2165
https://ossci-raw-job-status.s3.amazonaws.com/log/11107038595 2023-02-04T09:07:32.8667463Z hf_Bert 1.2181
https://ossci-raw-job-status.s3.amazonaws.com/log/11107003548 2023-02-04T09:01:23.7997973Z hf_Bert 1.2093
https://ossci-raw-job-status.s3.amazonaws.com/log/11106940460 2023-02-04T08:53:36.2376525Z hf_Bert 1.2137
https://ossci-raw-job-status.s3.amazonaws.com/log/11106155151 2023-02-04T06:59:28.1053159Z hf_Bert 1.2148
https://ossci-raw-job-status.s3.amazonaws.com/log/11105798189 2023-02-04T06:51:33.4027092Z hf_Bert 1.2094
https://ossci-raw-job-status.s3.amazonaws.com/log/11105322003 2023-02-04T06:36:44.2477712Z hf_Bert 1.2172
https://ossci-raw-job-status.s3.amazonaws.com/log/11105008449 2023-02-04T06:24:21.6702031Z hf_Bert 1.2064
https://ossci-raw-job-status.s3.amazonaws.com/log/11105008315 2023-02-04T06:04:42.4172887Z hf_Bert 1.2128
https://ossci-raw-job-status.s3.amazonaws.com/log/11104835835 2023-02-04T05:56:50.2969724Z hf_Bert 1.208
https://ossci-raw-job-status.s3.amazonaws.com/log/11104413324 2023-02-04T05:37:33.8504218Z hf_Bert 1.2114
https://ossci-raw-job-status.s3.amazonaws.com/log/11104353580 2023-02-04T04:40:53.4400698Z hf_Bert 1.2071
https://ossci-raw-job-status.s3.amazonaws.com/log/11103622770 2023-02-04T01:34:59.8147457Z hf_Bert 1.214
https://ossci-raw-job-status.s3.amazonaws.com/log/11102256895 2023-02-03T23:44:22.6601651Z hf_Bert 1.2087
https://ossci-raw-job-status.s3.amazonaws.com/log/11101942372 2023-02-03T23:22:47.1968593Z hf_Bert 1.2061

@weiwangmeta weiwangmeta added the ciflow/trunk Trigger trunk jobs on your pull request label Feb 6, 2023
desertfire
desertfire approved these changes Feb 6, 2023
View reviewed changes
Copy link
Contributor

@desertfire desertfire left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM. Please keep close eyes on this test after landing to see if there is any flakiness.

@weiwangmeta
Copy link
Contributor Author

@pytorchbot merge -f "#94067 would fix the one & only inductor non-deterministic sebotnet error".

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@weiwangmeta weiwangmeta mentioned this pull request Feb 7, 2023
Closed
@weiwangmeta weiwangmeta deleted the weiwangmeta/bump_smoketest_threshold branch February 10, 2023 03:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@desertfire desertfire desertfire approved these changes

@ngimel ngimel Awaiting requested review from ngimel

Assignees

No one assigned

Labels

ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request Merged module: dynamo topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@weiwangmeta @pytorchmergebot @desertfire