Optimize ReduceSumFloatCudaKernel with GEMM

[liujuncheng]

No description provided.

[github-actions]

Speed stats:

GPU Name: GeForce GTX 1080 

✔️ OneFlow resnet50 time: 128.3ms (= 12828.5ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 138.5ms (= 13848.1ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.08 (= 138.5ms / 128.3ms)

✔️ OneFlow resnet50 time: 77.7ms (= 7770.8ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 83.9ms (= 8390.6ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.08 (= 83.9ms / 77.7ms)

OneFlow resnet50 time: 49.1ms (= 9828.7ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 57.6ms (= 11524.7ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.17 (= 57.6ms / 49.1ms)

OneFlow resnet50 time: 40.3ms (= 8055.5ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 46.7ms (= 9340.4ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.16 (= 46.7ms / 40.3ms)

OneFlow resnet50 time: 39.5ms (= 7892.6ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 39.0ms (= 7794.2ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 0.99 (= 39.0ms / 39.5ms)

✔️ OneFlow resnet50 time: 142.1ms (= 14214.9ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 163.8ms (= 16379.0ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 163.8ms / 142.1ms)

OneFlow resnet50 time: 89.1ms (= 8914.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 106.8ms (= 10677.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.20 (= 106.8ms / 89.1ms)

OneFlow resnet50 time: 59.3ms (= 11851.9ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 75.7ms (= 15138.5ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.28 (= 75.7ms / 59.3ms)

OneFlow resnet50 time: 53.3ms (= 10666.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 69.3ms (= 13860.7ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.30 (= 69.3ms / 53.3ms)

OneFlow resnet50 time: 47.1ms (= 9420.5ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 64.6ms (= 12916.5ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.37 (= 64.6ms / 47.1ms)