TF-TRT Dynamic Shapes Feature Tracker

[tfeher]

Introduction

The dynamic shape mode in TF-TRT utilizes TensorRT’s dynamic shape feature to improve the conversion rate of networks and handle networks with unknown input shapes efficiently. This issue tracks the ongoing development to enable TRT's dynamic shape mode trough TF-TRT.

Who will benefit with this feature?
The conversion rate and therefore the performance will improve for the following inference problems:

• Network with unknown input shapes (e.g. fully convolutional object detection networks)
• Networks where the first (batch) dimension of a tensor changes within the graph (e.g. BERT)
• Networks that have subgraphs where the tensors have non identical first dimension

Additionally the memory usage will improve: to handle input tensors with different shapes (eg image size, sequence length) currently requires separate TRT engine creation for each input. With dynamic shape mode a single engine can handle various input shapes.

Will this change the current api? How?

Some change in the conversion parameters will be necessary to enable/disable dynamic shape mode and provide a way to select optimization profiles.

Phase 1

The first phase of this work is the basic scaffolding to enable the TF-TRT converter to use TRT's dynami shape API.

0. Add implicit batch experimental Add implicit batch experimental #34293
1. Enable explicit batch mode Enable TF-TRT explicit batch mode #36379
2. Improve binding index query Improve TensorRT binding index query #36434
3. Add binding size specification Add TensorRT binding size dimension specification #36435
4. Define networks with dynamic shapes Define TensorRT network with dynamic shapes #36439
5. Add optimizaton profiles Add TensorRT optimization profiles #36660
6. Execution context managment Execution context management for TensorRT profiles #36664
7. TensorRT profile generation mode TensorRT profile generation mode #36729

Phase 2

Enable dynamic shape mode for ops used in MobileNet, ResNet, Bert. This includes improvement in the converters plus increasing their unit test coverage. Note that at this stage dynamic shape mode is still experimental.

• Refactor ExecuteTrtEngine Refactor ExecuteTrtEngine. #38118
• Use Unified Memory in TRT opconverter tests Use Unified Memory in TRT opconverter tests #38124
• ConvertSqueeze Test Squeeze op converter in dynamic shape mode #38146
• ConvertUnary, ConvertRsqrt TF-TRT test ConvertUnary in dynamic shape mode. #39153
• ConvertTranspose TF-TRT ConvertTranspose in dynamic shape mode #39151
• ConvertActivation, ConvertLeakyRelu TF-TRT test activation converter #39155
• BiasAdd - ConvertBiasAdd TF-TRT enable BiasAdd op in dynamic shape mode #39156
• Conv2D, DepthWiseConv2dNative TF-TRT Conv2d op conversion dynamic shape mode #39204
• ConvertConv2dBackPropInput TF-TRT Enable Conv2DBackpropInput conversion in explicit batch mode #47840
• ConvertPack TF-TRT Pack op conversion in dynamic shape mode #39859
• ConvertReshape TF-TRT Improve reshape op converter #40545
• ConvertExpandDims TF-TRT ExpandDims converter dynamic shape mode #39282
• ConvertSlice, ConvertStridedSlice TF-TRT Slice op converters explicit batch mode #40736
• ConvertGather TF-TRT GatherV2 op conversion dynamic shape mode #39848
• ConvertMatMul and ConvertBatchMatMul TF-TRT Improve matrix multiplication conversion and enable dynamic shape mode #47215
• ConvertBinary TF-TRT test Binary op conversion in dynamic shape mode  #39785
• ConvertSquaredDifference TF-TRT test SquaredDiff op conversion in dynamic shape mode #39758
• ConvertReduce TF-TRT reduction op converter tests #40201
• ConvertFusedBatchNorm TF-TRT test FusedBatchNorm op converter #40179
• ConvertSoftMax TF-TRT Dynamic shape mode test for Softmax op converter #47039
• ConvertShape Add TF-TRT converter for Shape op #39990
• ConvertPool TF-TRT Improve test coverage of pool op converters #40184
• Update related python integration tests: Unary_test.py, Batch_matmul_test.py, biasadd_matmul_test.py, Conv2d_test.py, reshape_traspose_test.py

Phase 3

This is a direct continuation of Phase 2. Ensure that all op converter support dynamic shape mode and test it. We have almost 20 converters to update and test. The bulk of this work is improving the test coverage.

• ConvertSquare [TFTRT] Add Dynamic Shape Testing for ConvertSquare #40483
• ConvertClipByValue [TFTRT - Dynamic Shape Phase 3] Add Dynamic Shape Testing for ConvertClipByValue #45589
• ConvertPad TF-TRT ConvertPad in dynamic shape mode #45597
• ConvertResize [TFTRT - Dynamic Shape Phase 3] Add Dynamic Shape Testing for ConvertResize & Various TF2TRT Node Convert Unittest Improvements #46376
• ConvertArgMinMax [TFTRT - Dynamic Shape Phase 3] Add Dynamic Shape Testing for ConvertArgMinMax #45862
• ConvertQuantize TF-TRT Test ConvertQuantize in dynamic shape mode #45599
• ConvertTopK TF-TRT Test ConvertTopK in dynamic shape mode #46299
• ConvertUnpack [TFTRT] Add Dynamic Shape Testing and fix Explicit Batch Mode for ConvertUnpack #48049
• ConvertAddN [TFTRT - Dynamic Shape Phase 3] Add Dynamic Shape Testing for ConvertAddN and DebugString templated for nested numerical vectors #46675
• ConvertConcat TF-TRT Test ConvertConcat in dynamic shape mode #46382
• ConvertConv3D [TFTRT] Add Dynamic Shape Testing for ConvertConv3D #46940
• ConvertDepthSpaceShuffle TF-TRT SpaceToDepth and DepthToSpace op converters for dynamic shape mode. #47590
• ConvertSplit [TF:TRT] ConvertSplit for dynamic shapes. #48246
• ConvertCombinedNMS [TF-TRT] Fix TopK Combined NMS #40062
• Enable calibration if static input shapes are used [TF-TRT] Enable INT8 calibration for use_implicit_batch=false when there is no dynamic shape inputs #48244

Phase 3+

Some converters in phase 3 were updated only for explicit batch support with static shape. Enable dynamic shape mode for them:

• ConvertCombinedNMS
• ConvertResize [TF:TRT] Enable Dynamic Shape Support for ConvertResize #51462
• ConvertStridedSlice [TF:TRT] Enable dynamic shape for "slice" and "strided slice" operations. #51475
• ConvertConv2DBackpropInput [TF:TRT] Enable dynamic batch dim for Conv2dBackpropInput #51468

Additionally:

• Fix shape output bug Shape output bug in dynamic shape mode tensorrt#251
  [TF:TRT] Always set binding dimension #52181
  [TF:TRT] Create execution context with device memory if shape output is present in TRT 7 #52186
• Update python integration tests Add TF-TRT Python integration tests in dynamic shape mode #51411 TF-TRT MNIST test with V2 converter #51471

Phase 4

Implement calibration in dynamic shape mode. Using TRT 7.1 one can run calibration in dynamic shape mode.

• TRTEngineOp: allow profile collection before calibration
• Refine APIs: build mode + calibration + lazy calibration

Phase 5

Test performance of dynamic shape mode

Phase 6

Define API to enable dynamic shape and specify optimization profiles.

• Finalize API - implemeted here
• Implement additional optimization profiles
  • basic profiles for testing Add TF-TRT optimization profiles for testing #45588
  • Pass profile params to TrtShapeOptimizationProfiles class TF-TRT Prefer static shapes #48060
  • Implement Range, Range+Optimal profiles TF-TRT implement Range and RangeOptimal optimization profile strategies #48414

Phase 7

• C++ conversion API TF-TRT C++ conversion #52012
  Optional elements from Phase 6 are moved here.
• Implement UserDefined profile
• Change default conversion param from implicit batch mode to dynamic shape mode

Tagging @DEKHTIARJonathan and @bixia1

[DrXuQian]

Hi tfeher, About 'Networks where the first (batch) dimension of a tensor changes within the graph (e.g. BERT)'.
Why would the first dimension change within the graph? I would assume that once the batch size is determined, the shape won't change throughout graph of BERT.

[tfeher]

Why would the first dimension change within the graph?

Some networks do reshape operations that change the first dim. It is done internally to express some operations more conveniently, the output is usually reshaped again to have the expected batch size. An example is BERT TF1 model.

The dynamic shape feature of TF-TRT improves the TRT conversion of such networks.

[jiweibo]

Hi, @tfeher can tf-trt dynamic_batch support the situation of being divided into multiple trt subgraphs now?
And how should we set the shape information of the min, max, and opt of the internal subgraph?

[tfeher]

can tf-trt dynamic_batch support the situation of being divided into multiple trt subgraphs now?

Yes, dynamic shape mode supports graphs that have multiple TRT subgraphs. There is a known issue tensorflow/tensorrt#251, which occurs if the trt_engine_op is trying to output shape tensors. Otherwise it should work.

And how should we set the shape information of the min, max, and opt of the internal subgraph?

You only need to set the shape information for the inputs of the model. From these we calculate the size of any tensor in the graph, and set input shape information for the internal TRT subgraphs (trt_engine_ops).

[christopherbate]

PR for Conv2dBackpropInput #51468