Questions about the Reformat layer

[ZhangGe6]

Hello, I am using TensorRT on Nvidia A100, and using the fantastic trex to visualize the engine graph.

When I build the engine from resnet50, I get the plan graph with Reformat layers near both input and output ends (Fig. a). Because I only care about the output of the formal part of resnet50 (onnx model link), I remove the other nodes and rebuild the engine again. However, the Reformat layers disappear (Fig. b).

So I am very curious about:

1. Why are the reformat layers inserted (in Fig. a)? I notice that the Reformat layer changes the tensor format from NCHW and NC/4HW4 for the following convolution layers. If the Reformat layers help accelerate inference, can you share some explanations/materials?

   • I have found the layout illustration in the Nvidia official doc, but still a little confused.

2. Is there a method to keep the Reformat layer inserting behavior consistent (between Fig. a & Fig.b)? That is to say, if I want to insert Reformat layers for the engine in Fig. b (It's better if the inserted Reformat layers are the same as the ones in Fig. a), how should I do?

Thanks in advance!

[zerollzeng]

Reformat layer is inserted automatically by TRT for performance or other considerations, so users don't need to worry about them, they already are in best-perf-config.

let's take a simple example, suppose I have an ONNX model that has only 1 conv, and by default the input and output are NCHW, there are many conv kernels implemented in TRT, e.g. if the normal conv takes NCHW input and produces NCHW output has a perf of 100ms, but another kernel that can take NCHW input and NC/4HW4 output has a perf of 30ms, and the NC/4HW4 -> NCHW reformat only take 40ms, because 30 + 40 < 100. so TRT will choose the global best config which is the latter.

[ZhangGe6]

Got it. Thanks!