Optimize Gather CUDA Kernel

[liujuncheng]

No description provided.

[github-actions]

Speed stats:

GPU Name: GeForce GTX 1080 

OneFlow resnet50 time: 136.7ms (= 13673.7ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 140.8ms (= 14077.8ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.03 (= 140.8ms / 136.7ms)

OneFlow resnet50 time: 78.5ms (= 7847.6ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 85.2ms (= 8521.5ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.09 (= 85.2ms / 78.5ms)

OneFlow resnet50 time: 55.6ms (= 11126.5ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 56.8ms (= 11369.6ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.02 (= 56.8ms / 55.6ms)

OneFlow resnet50 time: 43.1ms (= 8617.9ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 47.4ms (= 9486.2ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.10 (= 47.4ms / 43.1ms)

OneFlow resnet50 time: 40.4ms (= 8083.1ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 39.0ms (= 7795.1ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 0.96 (= 39.0ms / 40.4ms)

OneFlow resnet50 time: 148.2ms (= 14822.6ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 158.2ms (= 15819.2ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.07 (= 158.2ms / 148.2ms)

OneFlow resnet50 time: 89.9ms (= 8988.3ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 98.8ms (= 9876.2ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.10 (= 98.8ms / 89.9ms)

OneFlow resnet50 time: 64.0ms (= 12809.5ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 73.3ms (= 14660.4ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.14 (= 73.3ms / 64.0ms)

OneFlow resnet50 time: 59.8ms (= 11969.6ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 62.7ms (= 12549.8ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.05 (= 62.7ms / 59.8ms)

OneFlow resnet50 time: 56.3ms (= 11260.9ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 56.0ms (= 11193.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 0.99 (= 56.0ms / 56.3ms)