Fix perf regression in ShuffleRows #5417
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RowShufflingTransformer is using ChannelReader incorrectly. It needs to block waiting for items to read and was Thread.Sleeping in order to wait, but not spin the current core. This caused a major perf regression. The fix is to block synchronously correctly - by calling AsTask() on the ValueTask that is returned from the ChannelReader and block on the Task. Fix dotnet#5416
stephentoub
approved these changes
Oct 5, 2020
Codecov Report
@@ Coverage Diff @@
## master #5417 +/- ##
==========================================
- Coverage 74.09% 74.01% -0.08%
==========================================
Files 1019 1020 +1
Lines 190363 190375 +12
Branches 20469 20471 +2
==========================================
- Hits 141047 140913 -134
- Misses 43788 43916 +128
- Partials 5528 5546 +18
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
@eerhardt Thanks for helping with this issue. |
frank-dong-ms-zz
added a commit
that referenced
this pull request
Oct 8, 2020
* Update to Onnxruntime 1.5.1 (#5406) * Added variables to tests to control Gpu settings * Added dependency to prerelease * Updated to 1.5.1 * Remove prerelease feed * Nit on GPU variables * Change the _maxCalibrationExamples default on CalibratorUtils (#5415) * Change the _maxCalibrationExamples default * Improving comments * Fix perf regression in ShuffleRows (#5417) RowShufflingTransformer is using ChannelReader incorrectly. It needs to block waiting for items to read and was Thread.Sleeping in order to wait, but not spin the current core. This caused a major perf regression. The fix is to block synchronously correctly - by calling AsTask() on the ValueTask that is returned from the ChannelReader and block on the Task. Fix #5416 Co-authored-by: Antonio Velázquez <38739674+antoniovs1029@users.noreply.github.com> Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>
mstfbl
pushed a commit
to mstfbl/machinelearning
that referenced
this pull request
Nov 11, 2020
RowShufflingTransformer is using ChannelReader incorrectly. It needs to block waiting for items to read and was Thread.Sleeping in order to wait, but not spin the current core. This caused a major perf regression. The fix is to block synchronously correctly - by calling AsTask() on the ValueTask that is returned from the ChannelReader and block on the Task. Fix dotnet#5416
mstfbl
pushed a commit
that referenced
this pull request
Nov 11, 2020
RowShufflingTransformer is using ChannelReader incorrectly. It needs to block waiting for items to read and was Thread.Sleeping in order to wait, but not spin the current core. This caused a major perf regression. The fix is to block synchronously correctly - by calling AsTask() on the ValueTask that is returned from the ChannelReader and block on the Task. Fix #5416
mstfbl
pushed a commit
to mstfbl/machinelearning
that referenced
this pull request
Nov 12, 2020
* Update to Onnxruntime 1.5.1 (dotnet#5406) * Added variables to tests to control Gpu settings * Added dependency to prerelease * Updated to 1.5.1 * Remove prerelease feed * Nit on GPU variables * Change the _maxCalibrationExamples default on CalibratorUtils (dotnet#5415) * Change the _maxCalibrationExamples default * Improving comments * Fix perf regression in ShuffleRows (dotnet#5417) RowShufflingTransformer is using ChannelReader incorrectly. It needs to block waiting for items to read and was Thread.Sleeping in order to wait, but not spin the current core. This caused a major perf regression. The fix is to block synchronously correctly - by calling AsTask() on the ValueTask that is returned from the ChannelReader and block on the Task. Fix dotnet#5416 Co-authored-by: Antonio Velázquez <38739674+antoniovs1029@users.noreply.github.com> Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>
mstfbl
pushed a commit
that referenced
this pull request
Nov 12, 2020
* Update to Onnxruntime 1.5.1 (#5406) * Added variables to tests to control Gpu settings * Added dependency to prerelease * Updated to 1.5.1 * Remove prerelease feed * Nit on GPU variables * Change the _maxCalibrationExamples default on CalibratorUtils (#5415) * Change the _maxCalibrationExamples default * Improving comments * Fix perf regression in ShuffleRows (#5417) RowShufflingTransformer is using ChannelReader incorrectly. It needs to block waiting for items to read and was Thread.Sleeping in order to wait, but not spin the current core. This caused a major perf regression. The fix is to block synchronously correctly - by calling AsTask() on the ValueTask that is returned from the ChannelReader and block on the Task. Fix #5416 Co-authored-by: Antonio Velázquez <38739674+antoniovs1029@users.noreply.github.com> Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>
ghost
locked as resolved and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
RowShufflingTransformer is using ChannelReader incorrectly. It needs to block waiting for items to read and was Thread.Sleeping in order to wait, but not spin the current core. This caused a major perf regression.
The fix is to block synchronously correctly - by calling AsTask() on the ValueTask that is returned from the ChannelReader and block on the Task.
Fix #5416
Results of the added benchmark:
cc @jwood803 @ogo-adp @stephentoub