[CPUBackend] Conv kernel performance regression

[jfix71]

I've tracked down at least one commit that has contributed to performance regression for the conv kernel: a2db2f7. E.g. this reduced Resnet50 FPS from 13.2 to 8.3 on my laptop.

Here is the profile for good perf of 100 inferences prior to the change:

And bad perf after the change:

The specialized libjit_convDKKC8_f_44_specialized seems to be one major problem area:

; Function Attrs: noinline
define internal void @libjit_convDKKC8_f_44_specialized(float*, float*, float*, float*, i64*, i64*, i64*, i64*, i64, i64, i64, i32, i32, i32, i32) #8 {
entry:
  call void @libjit_convDKKC8_f(float* %0, float* %1, float* %2, float* %3, i64* getelementptr inbounds ([4 x i64], [4 x i64]* @26, i32 0, i32 0), i64* getelementptr inbounds ([4 x i64], [4 x i64]* @26, i32 0, i32 0), i64* getelementptr inbounds ([5 x i64], [5 x i64]* @27, i32 0, i32 0), i64* getelementptr inbounds ([1 x i64], [1 x i64]* @12, i32 0, i32 0), i64 3, i64 1, i64 1, i32 0, i32 2, i32 5, i32 4)
  ret void
}

Which corresponds to:

filterSize: 3
stride: 1
pad: 1
pixelScanFirst: 0
numDepthRegs: 2
sizeGroupY: 5
depthStrips: 4

[nadavrot]

I can reproduce the regression locally on my mac.

I wonder if things are slower simply because the bug fix makes us not skip the last y-group. One way to find out would be to comment out the code that starts with "Handle the remaining Y in the row in groups of size 1."

@jfix71 Do you know what are the (x,y) dimensions of the convolutions that are hit the most?

[jfix71]

I wonder if things are slower simply because the bug fix makes us not skip the last y-group. One way to find out would be to comment out the code that starts with "Handle the remaining Y in the row in groups of size 1."

Yep, I commented out that if block and the performance regression seems to disappear.

Do you know what are the (x,y) dimensions of the convolutions that are hit the most?

Here are the number of times each input HxW is seen:

14x14:  190
28x28:  130
56x56:  130
7x7:     70
224x224: 10

[nadavrot]

The current implementation uses 2x5 register blocking. In theory a 3x4 or 4x3 register blocking should be faster because it has better compute/memory ratio. I suspect that the perf gains from 2x5 were related to the bug where we skipped a big row of y. We should try to experiment with the 3x4 or 4x3 kernels again.

[yzhao30]

I did some investigation.

1. I can reproduce the issue. The FPS performance for resnet50 is about 13 now and after commenting out the block, it restores to around 8.

2. I use a 3x4 kernel but it raises the segment fault error.

resnet50:convregress* λ env QUANTIZE=NO make
mkdir -p build
clang++ -std=c++11 -c -g resnet50.cpp -o build/main.o
for file in predict_net.pbtxt predict_net.pb init_net.pb; do \
		wget http://fb-glow-assets.s3.amazonaws.com/models/resnet50/$file -P resnet50 -nc; \
	done
File ‘resnet50/predict_net.pbtxt’ already there; not retrieving.

File ‘resnet50/predict_net.pb’ already there; not retrieving.

File ‘resnet50/init_net.pb’ already there; not retrieving.

mkdir -p build
~/Documents/glow_release/bin/image-classifier ~/Documents/glow/tests/images/imagenet/dog_207.png -image_mode=0to1 -m resnet50 -model_input_name=gpu_0/data -cpu -emit-bundle build -g
clang++ -o build/resnet50 build/resnet50.o build/main.o -lpng
cd build; \
	for file in ~/Documents/glow/tests/images/imagenet/*; do \
		./resnet50 $file; \
	done
Allocated weights of size: 102121984
Expected weights of size: 102121984
Loaded weights of size: 102121984 from the file resnet50.weights
Allocated mutable weight variables of size: 606144
Loaded image: /Users/yzhao30/Documents/glow/tests/images/imagenet/cat_285.png
Loaded images size in bytes is: 602112
Copying image data into mutable weight vars: 602112 bytes
/bin/sh: line 1: 91343 Segmentation fault: 11  ./resnet50 $file
Allocated weights of size: 102121984
Expected weights of size: 102121984
Loaded weights of size: 102121984 from the file resnet50.weights
Allocated mutable weight variables of size: 606144
Loaded image: /Users/yzhao30/Documents/glow/tests/images/imagenet/dog_207.png
Loaded images size in bytes is: 602112
Copying image data into mutable weight vars: 602112 bytes
/bin/sh: line 1: 91344 Segmentation fault: 11  ./resnet50 $file
Allocated weights of size: 102121984
Expected weights of size: 102121984
Loaded weights of size: 102121984 from the file resnet50.weights
Allocated mutable weight variables of size: 606144
Loaded image: /Users/yzhao30/Documents/glow/tests/images/imagenet/zebra_340.png
Loaded images size in bytes is: 602112
Copying image data into mutable weight vars: 602112 bytes
/bin/sh: line 1: 91345 Segmentation fault: 11  ./resnet50 $file
make: *** [run] Error 139

3. I use a 4x3 kerne and the performance seems worse
   

[bertmaher]

As far as I can tell we concluded that this performance regression was due to a necessary correctness fix and that there wasn't an obvious way to reclaim the perf. I'm going to close this particular issue since "improve the performance of convolutions" is a never-ending task :-).