[To Disable LayerFusion] add copyOutputToHost function for only selected outputs #495
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nvpohanh
requested changes
Mar 16, 2023
| //! | ||
| //! \brief Copy the selected contents of output device buffers by bidingIdx to output host buffers synchronously. | ||
| //! | ||
| void copyOutputToHost(const std::vector<int>& bindingIndices) |
There was a problem hiding this comment.
Thanks for the contribution!
To follow our coding convention, it would be great if you can make the following modifications:
- Use east-const style (i.e.
const std::vector<int>&->std::vector<int> const&) for all occurrences ofconst. - Use
int32_tinstead ofint. - Use 4-whitespace indentation instead of 2-whitespace indentation.
We appreciate it!
| const cudaMemcpyKind memcpyType = cudaMemcpyDeviceToHost; | ||
| if (!mEngine->bindingIsInput(i)) | ||
| { | ||
| CHECK(cudaMemcpy(dstPtr, srcPtr, byteSize, memcpyType, stream)); |
There was a problem hiding this comment.
cudaMemcpy should be cudaMemcpyAsync
| { | ||
| for (int i : bindingIndices) | ||
| { | ||
| void* dstPtr = mManagedBuffers[i]->hostBuffer.data(); |
There was a problem hiding this comment.
Could we move these inside the if(){....} clause?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When there is an bug(#380) in layer fusion or you just want to debug layer fusions,
using mark_output is the only solution to disable layer fusion now(#252).
https://github.com/NVIDIA/TensorRT/issues/252#issuecomment-577468499
But the latency increases because to use copyOutputToHost brings unintended outputs from device to host.
https://github.com/NVIDIA/TensorRT/issues/252#issuecomment-593716265
Therefore, added two functions in buffers.h to bring the selected outputs only(by index) to host.