[Topi, ARM] Disbale Winograd for quantized tensors.

[anijain2305]

Winograd Conv2D on integer tensors lead to bad accuracy. I think this is due to underlying maths that needs floating point multiplication or high precision computation. Enabling it only for FP32 tensors.

@masahi @FrozenGene @zhen-jia @u99127

Without winograd, the TOP 5 labels for few images with class 0 label is

1 [  0 389 391 758 394]
2 [390 394  48 103   0]
3 [  0 389 241 240  28]
4 [133 124 132 131 554]
5 [  0 389 390 939 391]
6 [  0 389  37 997 241]
7 [  0 758 389 391 696]
8 [  0 389 838 930 393]
9 [  0 389   1 395 758]
10 [  0 391 758 389 921]
11 [  0 389   1 391 395]
12 [  0 241 955 944 240]
13 [  0 389   5 393  57]
14 [758 391 346 268 395]
15 [  0 389   1 838 393]
16 [  0 389 241 838  28]
17 [  0 389   1 395  37]

With winograd, the TOP 5 labels for few images with class 0 label is

1 [904 818 828 556 648]
2 [904 632 482 270 556]
3 [904 556 270 632 650]
4 [904 828 650 556 383]
5 [904 632 270 556 482]
6 [904 650 828 556 270]
7 [904 818 650 270 249]
8 [904 632 818  60 482]
9 [904 632 828 556 270]
10 [904 818 556 650 270]
11 [904 632 482 556 270]
12 [904 828 632 482 270]
13 [904 632 818 270 556]
14 [904 828 650 818 556]
15 [904 632 482  60 818]

Essentially, accuracy goes down to 0%

[masahi]

Do you have concrete data? Say accuracy difference on imagenet with and without winograd.

[anijain2305]

Output is all zero. It is not doing the right compute anymore, so it 0% accuracy.

[anijain2305]

It might be because of this
https://github.com/apache/incubator-tvm/blob/c936a81dab2b4b4b595d02153a6654b9d4e09cd5/topi/python/topi/nn/winograd_util.py#L83-L131

The numbers here need to FP32

[masahi]

I will take a look at how MKLDNN does winograd on int8.

I also wonder if weight quantization is working correctly. Since transformed weight shape is not in (OC, OI, KH, KW) format, maybe per channel quantize is broken if winograd is not taken into acount.

UPDATE: Since weight quantization happens before winograd transform, it is not a problem.

[masahi]

Seems they cast inputs to fp32 and does multiply in fp32
https://github.com/oneapi-src/oneDNN/blob/2994eae18d84d4c7eab2f1402444b91bffa140a9/src/cpu/jit_avx512_core_u8s8s32x_wino_convolution.cpp#L149-L153

Transform coefficients they use
https://github.com/oneapi-src/oneDNN/blob/2994eae18d84d4c7eab2f1402444b91bffa140a9/src/cpu/jit_avx512_core_u8s8s32x_wino_convolution.cpp#L43-L44

[anijain2305]

Thanks for sharing the pointers @masahi
That makes sense. It also explains why accuracy is bad in master. We do not cast back to FP32.

Maybe we can handle that in a separate PR.

[masahi]

If the only missing bits is the cast to fp32, I hope our perf result is still valid...

[cbalint13]

Float16 Too ? (perhaps double aka float64 too)
I get good results with it (e.g yolov3-tiny on mali).
Wouldn't be better to deny int8/int16/int32 ?

[cbalint13]

It might be because of this
https://github.com/apache/incubator-tvm/blob/c936a81dab2b4b4b595d02153a6654b9d4e09cd5/topi/python/topi/nn/winograd_util.py#L83-L131

The numbers here need to FP32

degree is an integer (fits in INT8) that requests filter size (usually 3x3), it will also reflect the degree of polynomial.

[cbalint13]

Thanks for sharing the pointers @masahi
That makes sense. It also explains why accuracy is bad in master. We do not cast back to FP32.

Maybe we can handle that in a separate PR.

Casting into FP is just a workaround.
Winograd on INT is doable but is tricky.
One wise approach is: https://arxiv.org/abs/2003.08646

[cbalint13]

Output is all zero. It is not doing the right compute anymore, so it 0% accuracy.

I think operations overflows badly (untested, its just supposition) leading to lots of 0 MUL ops.
I happens that lot of INT8 ops goes to INT8 (overflowing)

[masahi]

A related discussion at andravin/wincnn#16

It might be better to do weight transform before weight quantization. But given the order of Relay passes, I don't see how that can be achieved.

[tqchen]

We might want to do winograd transformation explicitly in relay(as opposed to implicitly in topi), that would enable quantization after the transformation.

But for this particular PR, perhaps we could go ahead and disable i8 winograd for now

[anijain2305]

@cbalint13 @masahi PTAL again. I relaxed to have any type of floats. And also added the Top-5 labels for few images in the desciption.

[masahi]

Thanks @anijain2305 @cbalint13 @tqchen

We should follow up to have a proper quantized winograd for ARM and possibly for x86 too. I'm interested in this topic, but I don't have much time recently.

[chrschinab]

I currently face the issue that Winograd implementation for quantized int8 operators destroys accuracy (see my topic on discuss.tvm.ai) as described above.

Is the functionality to disable Winograd for quantized operators already implemented? How to do that?

[BUG1989]

@masahi In our project. The int8 winograd F(2,3) has the same accuracy as original int8 conv3x3s1, please refer it.
andravin/wincnn#16 (comment)
ncnn int8 pr
naive c code

[cbalint13]

@BUG1989 ,

Thanks for the share and links ! I was aware of that work, will try to add a schedule to TVM for int8 cases.