Avoid marking every profile loop stop as Collection stage, use data available to mark errored stages. #977
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This pull request was exported from Phabricator. Differential Revision: D61226939 |
aaronenyeshi
approved these changes
Aug 15, 2024
…vailable to mark errored stages. (pytorch#977) Summary: Pull Request resolved: pytorch#977 We already have the data collected to know if the collection was stopped due to `collectionDone` or `stopCollection`, the later is only set when CUPTI abruptly stops in events like not finding buffers. We infact also set this in the itnernal Error Counters, so leverage that functionality within UST logging as well to denote a terminal stage within Kineto. Reviewed By: aaronenyeshi Differential Revision: D61226939
|
This pull request was exported from Phabricator. Differential Revision: D61226939 |
sanrise
force-pushed
the
export-D61226939
branch
from
1ee8754
to
6314734
Compare
|
This pull request has been merged in 7d5e58f. |
staugust
pushed a commit
to staugust/kineto
that referenced
this pull request
Aug 27, 2024
…vailable to mark errored stages. (pytorch#977) Summary: Pull Request resolved: pytorch#977 We already have the data collected to know if the collection was stopped due to `collectionDone` or `stopCollection`, the later is only set when CUPTI abruptly stops in events like not finding buffers. We infact also set this in the itnernal Error Counters, so leverage that functionality within UST logging as well to denote a terminal stage within Kineto. Reviewed By: aaronenyeshi Differential Revision: D61226939 fbshipit-source-id: a4d5fa525d4457d44f0b959e4761b82de160152c
staugust
pushed a commit
to staugust/kineto
that referenced
this pull request
Aug 27, 2024
…vailable to mark errored stages. (pytorch#977) Summary: Pull Request resolved: pytorch#977 We already have the data collected to know if the collection was stopped due to `collectionDone` or `stopCollection`, the later is only set when CUPTI abruptly stops in events like not finding buffers. We infact also set this in the itnernal Error Counters, so leverage that functionality within UST logging as well to denote a terminal stage within Kineto. Reviewed By: aaronenyeshi Differential Revision: D61226939 fbshipit-source-id: a4d5fa525d4457d44f0b959e4761b82de160152c
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
We already have the data collected to know if the collection was stopped due to
collectionDoneorstopCollection, the later is only set when CUPTI abruptly stops in events like not finding buffers.We infact also set this in the itnernal Error Counters, so leverage that functionality within UST logging as well to denote a terminal stage within Kineto.
Differential Revision: D61226939