Int8 gemm weight pre-packing as `CUBLASLT_ORDER_COL32_2R_4R4`

[vadimkantorov]

As I understand, for max perf of int8 gemms, the weights should be prepacked as CUBLASLT_ORDER_COL32_2R_4R4 memory format and inputs as COL32.

At trex graph we can see that output of int8 gemm is formatted as COL32 (C32 means that COL32 layout was used, right?), but can't see if TensorRT prepacked the weights as 4r4. Does it prepack the weights as 4r4? Is TensorRT actually using COL32 for input/output of the quantized MatMul?

Thanks :)

[zerollzeng]

@nvpohanh ^ ^

[zerollzeng]

C32 means that COL32 layout was used, right?

I think it means NC/32HW32.

[zerollzeng]

As I understand, for max perf of int8 gemms, the weights should be prepacked as CUBLASLT_ORDER_COL32_2R_4R4 memory format and inputs as COL32.

I wonder whether this is always true? @nvpohanh can you shed some light here :-D

[vadimkantorov]

I think it means NC/32HW32.

well, what does NC/32HW32 mean then? :)

created a related issue on cublasLt tracing: #3251

[nvpohanh]

TRT already does all the weight pre-processing automatically to achieve the best perf.

[vadimkantorov]

And what about the inputs? Is COL32 used?

Btw, should CUBLASLT_LOG_FILE=cublaslt.log CUBLASLT_LOG_MASK=31 CUBLASLT_LOG_LEVEL=5 env variables work for primitive cublasLt call tracing inside trtexec --loadEngine?

[nvpohanh]

In this specific case, TRT is actually using a Convolution kernel (implicit_gemm) to compute the gemm and the expected input format is NCHW_INT8 for that kernel.

TRT does not use CUBLASLt, so I doubt those env var could work.

Is this a Transformer? If so, TRT doesn't have good perf with Transformer + implicit quantization. It is recommended to use explicit quantizations for Transformers.

[vadimkantorov]

Hmm, very strange. Yeah, it is a transformer. Why does it use convolution and not a MatMul here?

E.g. FasterTransformer takes care everywhere to properly use COL32/4R4. Is cublasLt achieving faster perf using cublasLt for int8 kernels and funky memory formats? Is it possible to somehow use FasterTransformer plugin to compare the perf (a simple quantized gemm and a simple quantized MLP within taken from a transformer model)? (related: #3230)

Does TRT use cudnn for this call (implicit gemm)? Seems also not, right? Then should CUDNN_LOGINFO_DBG=1 CUDNN_LOGDEST_DBG=cudnn.log help for basic tracing?

[nvpohanh]

TRT uses internal kernel library directly, which is also what CUBLAS/CUBLASLt/CUDNN use. TRT doesn't go through CUBLAS/CUDNN anymore.

Why does it use convolution and not a MatMul here?

It is because TRT's Transformer optimization path is not triggered in implicit quantization mode, so TRT treats this as a normal ConvNets. To trigger TRT's Transformer-specific optimizations, explicit quantization mode must be used. If triggered, TRT will use the COL32/4R4 format for the weights properly.

[vadimkantorov]

Well, the shapes are BTC and the matmul is done over the last dimension (part of MLP). Why would it not use MatMul?

Okay, got the point about explicit quantization! Any specific verbose logging lines we should be looking for to check if "transformer-specific optimizations" are used/triggered?

[nvpohanh]

Any specific verbose logging lines we should be looking for to check if "transformer-specific optimizations" are used/triggered?

Whenever you see ForeignNode in the trex plot