Dev add jacobian hessian test

[lihuizhao]

添加jacobian、hessian接口的测试代码

[github-actions]

View latest API docs preview at: https://oneflow-staging.oss-cn-beijing.aliyuncs.com/docs/Oneflow-Inc/oneflow/pr/10428/

[github-actions]

Speed stats:

GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 43.5ms (= 4350.5ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 58.1ms (= 5814.9ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.34 (= 58.1ms / 43.5ms)

OneFlow resnet50 time: 26.0ms (= 2601.1ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 39.4ms (= 3940.6ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.51 (= 39.4ms / 26.0ms)

OneFlow resnet50 time: 19.9ms (= 3982.9ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 36.0ms (= 7204.8ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.81 (= 36.0ms / 19.9ms)

OneFlow resnet50 time: 17.6ms (= 3511.6ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 32.2ms (= 6439.2ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.83 (= 32.2ms / 17.6ms)

OneFlow resnet50 time: 17.1ms (= 3423.4ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 29.0ms (= 5807.8ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.70 (= 29.0ms / 17.1ms)

OneFlow swin dataloader time: 0.199s (= 39.887s / 200, num_workers=1)
PyTorch swin dataloader time: 0.127s (= 25.468s / 200, num_workers=1)
Relative speed: 0.638 (= 0.127s / 0.199s)

OneFlow swin dataloader time: 0.055s (= 11.009s / 200, num_workers=4)
PyTorch swin dataloader time: 0.032s (= 6.480s / 200, num_workers=4)
Relative speed: 0.589 (= 0.032s / 0.055s)

OneFlow swin dataloader time: 0.030s (= 6.058s / 200, num_workers=8)
PyTorch swin dataloader time: 0.017s (= 3.314s / 200, num_workers=8)
Relative speed: 0.547 (= 0.017s / 0.030s)

❌ OneFlow resnet50 time: 49.0ms (= 4902.3ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 64.9ms (= 6490.5ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.32 (= 64.9ms / 49.0ms)

OneFlow resnet50 time: 36.7ms (= 3674.4ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 46.9ms (= 4687.9ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.28 (= 46.9ms / 36.7ms)

OneFlow resnet50 time: 28.1ms (= 5620.0ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 41.9ms (= 8379.1ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.49 (= 41.9ms / 28.1ms)

OneFlow resnet50 time: 24.9ms (= 4985.4ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 38.1ms (= 7616.4ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.53 (= 38.1ms / 24.9ms)

OneFlow resnet50 time: 24.2ms (= 4848.8ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 36.6ms (= 7320.8ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.51 (= 36.6ms / 24.2ms)

[github-actions]

View latest API docs preview at: https://oneflow-staging.oss-cn-beijing.aliyuncs.com/docs/Oneflow-Inc/oneflow/pr/10428/

[github-actions]

Speed stats:

GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 43.4ms (= 4340.0ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 57.2ms (= 5717.2ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.32 (= 57.2ms / 43.4ms)

OneFlow resnet50 time: 26.1ms (= 2610.5ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 37.4ms (= 3743.1ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.43 (= 37.4ms / 26.1ms)

OneFlow resnet50 time: 18.7ms (= 3737.7ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 37.0ms (= 7401.9ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.98 (= 37.0ms / 18.7ms)

OneFlow resnet50 time: 17.3ms (= 3453.1ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 30.6ms (= 6114.9ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.77 (= 30.6ms / 17.3ms)

OneFlow resnet50 time: 17.8ms (= 3569.6ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 29.5ms (= 5898.6ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.65 (= 29.5ms / 17.8ms)

OneFlow swin dataloader time: 0.201s (= 40.188s / 200, num_workers=1)
PyTorch swin dataloader time: 0.129s (= 25.713s / 200, num_workers=1)
Relative speed: 0.640 (= 0.129s / 0.201s)

OneFlow swin dataloader time: 0.054s (= 10.742s / 200, num_workers=4)
PyTorch swin dataloader time: 0.033s (= 6.567s / 200, num_workers=4)
Relative speed: 0.611 (= 0.033s / 0.054s)

OneFlow swin dataloader time: 0.031s (= 6.153s / 200, num_workers=8)
PyTorch swin dataloader time: 0.016s (= 3.289s / 200, num_workers=8)
Relative speed: 0.535 (= 0.016s / 0.031s)

❌ OneFlow resnet50 time: 49.2ms (= 4922.2ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 64.6ms (= 6461.6ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.31 (= 64.6ms / 49.2ms)

OneFlow resnet50 time: 36.7ms (= 3667.1ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 48.2ms (= 4817.6ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.31 (= 48.2ms / 36.7ms)

OneFlow resnet50 time: 28.5ms (= 5706.5ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 39.4ms (= 7871.6ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.38 (= 39.4ms / 28.5ms)

OneFlow resnet50 time: 25.1ms (= 5026.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 38.9ms (= 7780.2ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.55 (= 38.9ms / 25.1ms)

OneFlow resnet50 time: 23.8ms (= 4759.6ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 36.8ms (= 7363.8ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.55 (= 36.8ms / 23.8ms)