GPU memory utilization, operator execution modes #47
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: # New features: * Collects GPU memory utilization in addition to latency. This done using PyTorch's own memory tracker. * Added new modes of operator execution * DISCRETE: this is the original method. We time each operator execution individually. A synchronization point is inserted before and after each GPU operator launch. We also clear the cache between each execution. However this introduces various overhead that we may not observe when multiple operators execute sequentially. This is due to the async execution model of CUDA. * CONTINUOUS: In this mode, the operator is executed in a tight loop. We only time the overall loop time, then divide by the number of loop count to get the average operator execution time. The overheads are amortized over many iterations so it's closer to the actual kernel execution time. * CONTINUOUS_EVENTS: This is similar to CONTINUOUS mode, but we also insert before and after torch.cuda.Event in the loop. The operator time is calculated based on the delta of these events. This will give individual measurements, but may introduce minor overhead that's not observed in CONTINUOUS mode. **Note that the continuous mode is the same way to measure in compute/pt/ benchmarks. As shown below (2nd run), they produce the same result.** # A few to do items for the coming diffs: * convert to use the new torch nvtx D33155244 * be consistent using buck build run_batch script * nsys support Differential Revision: D34071165 fbshipit-source-id: 4713ecdcd2d5fdb6fd7b087748078166c28fb4dc
louisfeng
requested review from
srinivas212,
naghshineh and
kingchc
as code owners
facebook-github-bot
added
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D34071165 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
New features:
Note that the continuous mode is the same way to measure in compute/pt/ benchmarks. As shown below (2nd run), they produce the same result.
A few to do items for the coming diffs:
Differential Revision: D34071165