Speed up the training

[Yipeng1994]

Mentioned in https://github.com/Oneflow-Inc/OneTeam/issues/1735, some operators might need to be run as late as possible since they have a large activation time in cpu.
In this feature, we move those operators backward and reduce the idle time in cuda by 40% (11.5ms -> 7ms per iteration)

Currently no obvious speeding up

[github-actions]

Speed stats:

[github-actions]

Speed stats:

GPU Name: GeForce GTX 1080 









❌ OneFlow resnet50 time: 139.5ms (= 13950.6ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 160.3ms (= 16032.6ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.15 (= 160.3ms / 139.5ms)

OneFlow resnet50 time: 84.9ms (= 8491.9ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 108.3ms (= 10834.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.28 (= 108.3ms / 84.9ms)

OneFlow resnet50 time: 57.6ms (= 11529.5ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 77.3ms (= 15463.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.34 (= 77.3ms / 57.6ms)

OneFlow resnet50 time: 44.8ms (= 8961.5ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 69.6ms (= 13926.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.55 (= 69.6ms / 44.8ms)

OneFlow resnet50 time: 40.0ms (= 8000.5ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 64.6ms (= 12917.4ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.61 (= 64.6ms / 40.0ms)

[github-actions]

View latest API docs preview at: https://staging.oneflow.info/docs/Oneflow-Inc/oneflow/pr/9278/