Skip to content

Implementation of TRT wrapping via inference.json #620

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 38 commits into from
Sep 10, 2024
Merged

Implementation of TRT wrapping via inference.json #620

merged 38 commits into from
Sep 10, 2024

Conversation

borisfom
Copy link
Contributor

Sample implementation of using new trt_wrap() from MONAI
Depends on Project-MONAI/MONAI#7990

@borisfom
POC implementation of TRT wrapping via inference.json
85efcc3 
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
@borisfom borisfom mentioned this pull request Aug 16, 2024
Open
@borisfom
Copy link
Contributor Author

I observe 2.5x speedup with FP16 TRT for a small sample (28.57it/s vs 11it/s). Did not check the accuracy.

borisfom and others added 3 commits August 16, 2024 11:54
@borisfom
Bumped bundle version
98a615c 
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
@borisfom
Adding pathology_nuclei_classification and swin_unetr_btcv_segmentation
d3275e9 
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
18c580f 
for more information, see https://pre-commit.ci
Nic-Ma
Nic-Ma reviewed Aug 21, 2024
View reviewed changes
Nic-Ma
Nic-Ma reviewed Aug 21, 2024
View reviewed changes
borisfom and others added 5 commits August 21, 2024 19:11
@borisfom
Adjusting arguments
936103d 
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
@borisfom
Merge branch 'trt_wrapper' of github.com:borisfom/model-zoo into trt_…
58d76de 
…wrapper

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
@borisfom
Merge remote-tracking branch 'origin/dev' into trt_wrapper
9e4bac4 
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
@borisfom
Added vista3 inference_trt.json
8461033 
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
8588674 
for more information, see https://pre-commit.ci
yiheng-wang-nv
yiheng-wang-nv reviewed Aug 23, 2024
View reviewed changes
borisfom and others added 12 commits August 22, 2024 22:34
@borisfom
Renamed trt_wrap -> trt_compile
f7da0ec 
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
@borisfom
Merge branch 'trt_wrapper' of github.com:borisfom/model-zoo into trt_…
c8d64bf 
…wrapper

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
aab37c5 
for more information, see https://pre-commit.ci
@borisfom
Added TRT for pathology_nuclei_segmentation_classification
7b64083 
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
@borisfom
Merge branch 'trt_wrapper' of github.com:borisfom/model-zoo into trt_…
43da1f5 
…wrapper
@borisfom
Merge remote-tracking branch 'origin/dev' into trt_wrapper
3d8a738
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
4a9f2ba 
for more information, see https://pre-commit.ci
@borisfom
Merge remote-tracking branch 'origin/dev' into trt_wrapper
7496dfb
@borisfom
Added Vista2D TRT config
337fcf3 
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
@borisfom
Cleaned up using new config syntax, batch size added
16f720f 
Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
@borisfom
Merge branch 'trt_wrapper' of github.com:borisfom/model-zoo into trt_…
6cc423d 
…wrapper

Signed-off-by: Boris Fomitchev <bfomitchev@nvidia.com>
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
a205bf4 
for more information, see https://pre-commit.ci
@borisfom borisfom changed the title POC implementation of TRT wrapping via inference.json Implementation of TRT wrapping via inference.json Aug 27, 2024
@borisfom
Copy link
Contributor Author

borisfom commented Sep 6, 2024

@binliunls : I have merged origin/dev and fixed brats_mri_generative_diffusion and brats_mri_axial_slices_generative_diffusion. Note: for brats_mri_axial_slices_generative_diffusion, default
python -m monai.bundle run --config_file "['configs/inference.json', 'configs/inference_trt.json']" does not get to the diffusion forward, so no TRT is generated, and there may be issues with it. Let me know how to run it so that the diffusion actually gets executed.

@borisfom borisfom mentioned this pull request Sep 6, 2024
Closed
8 tasks
@binliunls
Copy link
Contributor

@binliunls : I have merged origin/dev and fixed brats_mri_generative_diffusion and brats_mri_axial_slices_generative_diffusion. Note: for brats_mri_axial_slices_generative_diffusion, default python -m monai.bundle run --config_file "['configs/inference.json', 'configs/inference_trt.json']" does not get to the diffusion forward, so no TRT is generated, and there may be issues with it. Let me know how to run it so that the diffusion actually gets executed.

Hi @borisfom ,
Thank you for your update. As shown in the inference config file, you should compile the diffusion and autoencoder with TRT instead of network, since the network parameter is not defined in either bundle. Please check if you can compile both diffusion and autoencoder.

Thanks,
Bin

@borisfom
Copy link
Contributor Author

borisfom commented Sep 6, 2024

@binliunls : Thanks, you are correct! Works as expected now.

@yiheng-wang-nv
update metadata
f0a9329 
Signed-off-by: Yiheng Wang <vennw@nvidia.com>
@yiheng-wang-nv
Copy link
Collaborator

/build

@yiheng-wang-nv
update mgpu test script
bde5d52 
Signed-off-by: Yiheng Wang <vennw@nvidia.com>
@yiheng-wang-nv
Copy link
Collaborator

/build

@binliunls
add benchmark for non generative models
c7c4b50 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
add benchmark results for generative bundles
626ba3c 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
add version for trt benchmark
7f47fb4 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
update vista2d version
5d58414 
Signed-off-by: binliu <binliu@nvidia.com>
@binliunls
update format
19ba83c 
Signed-off-by: binliu <binliu@nvidia.com>
@yiheng-wang-nv
Copy link
Collaborator

/build

@yiheng-wang-nv
use nvidia host large files
3e80c93 
Signed-off-by: Yiheng Wang <vennw@nvidia.com>
@yiheng-wang-nv
Copy link
Collaborator

/build

@yiheng-wang-nv
update vista2d
35e697c 
Signed-off-by: Yiheng Wang <vennw@nvidia.com>
@yiheng-wang-nv yiheng-wang-nv mentioned this pull request Sep 10, 2024
Closed
@yiheng-wang-nv
Copy link
Collaborator

/build

1 similar comment
@yiheng-wang-nv
Copy link
Collaborator

/build

@yiheng-wang-nv yiheng-wang-nv enabled auto-merge (squash) September 10, 2024 06:28
@yiheng-wang-nv yiheng-wang-nv merged commit efadc57 into Project-MONAI:dev Sep 10, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@KumoLiu KumoLiu KumoLiu left review comments

@Nic-Ma Nic-Ma Nic-Ma approved these changes

@yiheng-wang-nv yiheng-wang-nv yiheng-wang-nv approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@borisfom @KumoLiu @yiheng-wang-nv @Nic-Ma @binliunls