Skip to content

[Unity][MSC][M1.3] Add translate && codegen for tensorrt#15950

Merged
Hzfengsy merged 4 commits intoapache:unityfrom
Archermmt:msc
Oct 26, 2023
Merged

[Unity][MSC][M1.3] Add translate && codegen for tensorrt#15950
Hzfengsy merged 4 commits intoapache:unityfrom
Archermmt:msc

Conversation

@Archermmt
Copy link
Contributor

@Archermmt Archermmt commented Oct 19, 2023

This is a pull request for MSC(Multi-System Compile)
RFC: https://discuss.tvm.apache.org/t/rfc-unity-msc-introduction-to-multi-system-compiler/15251/5
Tracking issue: #15233

This is the 3rd part of Milestone 1: Finish RuntimeManager for frameworks

To limit each PR in reviewable size, the Milestone 1 will be split into some steps:
M1.1 Add translate && codegen for torch
M1.2 Add translate && codegen for tensorflow
M1.3 Add codegen for tensorrt
M1.4 Add Runner and test with relax
M1.5 Add Runner and test with torch
M1.6 Add Runner and test with tensorflow
M1.7 Add Runner and test with tensorrt

This PR support tensorrt in a codegen way. Same strategy is used in Nvidia Nackthon 2021: https://github.com/Archermmt/yolov3_dcn_nv_hackthon2021 (special price for latency and bug detection).

2 passes are added to support BYOC
1.BindShape: Works like FuseOpsByPattern(bind_constant=True). This pass bind the ShapeExpr of extern func to Expr.
2.FuseTuple: This pass fuse Tuple and TupleGetitem to byoc func.

Why use codegen flavor to support TensorRT:

  1. To enable deploy tensorrt solution in different runtime environment without re-compile the model.
  2. To enable the usage of MSC tools in TensorRT. Including pruning, debugging and so on.

cc @Hzfengsy @junrushao

@Hzfengsy Hzfengsy merged commit 5d62e9b into apache:unity Oct 26, 2023
@Archermmt Archermmt deleted the msc branch October 26, 2023 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants