Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev add jvp jacobian hessian #10412

Merged
merged 35 commits into from
Jan 23, 2024
Merged

Dev add jvp jacobian hessian #10412

merged 35 commits into from
Jan 23, 2024

Conversation

lihuizhao
Copy link
Contributor

分支工作

添加jvp、jacobian、hessian、hvp、vhp接口到funcitional.py文件,添加测试代码到test_autograd_functional.py文件

代码修改部分

  1. _construct_standard_basis_for(): 因fill_ 不支持 NonContiguous,将fill_赋值替换为循环赋值。
    (oneflow.torch.fill_()与oneflow.torch.diagonal()结合使用时结果错误 #10394)
  2. _construct_standard_basis_for(): 函数参数tensors: Tuple[torch.Tensor, ...]的写法会导致循环导入torch,因此将后面的描述去掉,变为tensors
  3. _grad_preprocess:因oneflow.torch没有is_sparse属性,只采用res.append(inp.view_as(inp))方式创建新tensor
  4. _jacfwd: 删除vectorize情况的代码
  5. 在test_autograd_functional.py文件中添加jvp、jacobian、hessian、hvp、vhp测试代码

添加接口

1704862723128

@lihuizhao lihuizhao requested a review from doombeaker as a code owner January 22, 2024 04:12
@marigoold marigoold enabled auto-merge (squash) January 22, 2024 04:59
@marigoold marigoold removed the request for review from doombeaker January 22, 2024 04:59
Copy link
Contributor

Copy link
Contributor

Speed stats:

@lihuizhao lihuizhao disabled auto-merge January 22, 2024 06:31
Copy link
Contributor

Copy link
Contributor

CI failed when running job: cpu-misc. PR label automerge has been removed

Copy link
Contributor

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

Copy link
Contributor

Copy link
Contributor

Copy link
Contributor

Copy link
Contributor

Speed stats:
GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 43.8ms (= 4380.7ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 57.5ms (= 5754.6ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.31 (= 57.5ms / 43.8ms)

OneFlow resnet50 time: 26.1ms (= 2611.1ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 37.7ms (= 3768.1ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.44 (= 37.7ms / 26.1ms)

OneFlow resnet50 time: 18.3ms (= 3662.4ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 35.0ms (= 6990.4ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.91 (= 35.0ms / 18.3ms)

OneFlow resnet50 time: 17.5ms (= 3508.0ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 31.2ms (= 6242.7ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 1.78 (= 31.2ms / 17.5ms)

OneFlow resnet50 time: 17.1ms (= 3413.6ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 28.4ms (= 5681.8ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.66 (= 28.4ms / 17.1ms)

OneFlow swin dataloader time: 0.200s (= 39.972s / 200, num_workers=1)
PyTorch swin dataloader time: 0.127s (= 25.351s / 200, num_workers=1)
Relative speed: 0.634 (= 0.127s / 0.200s)

OneFlow swin dataloader time: 0.056s (= 11.253s / 200, num_workers=4)
PyTorch swin dataloader time: 0.033s (= 6.537s / 200, num_workers=4)
Relative speed: 0.581 (= 0.033s / 0.056s)

OneFlow swin dataloader time: 0.030s (= 6.084s / 200, num_workers=8)
PyTorch swin dataloader time: 0.017s (= 3.312s / 200, num_workers=8)
Relative speed: 0.544 (= 0.017s / 0.030s)

❌ OneFlow resnet50 time: 49.9ms (= 4986.9ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 64.5ms (= 6454.6ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.29 (= 64.5ms / 49.9ms)

OneFlow resnet50 time: 36.4ms (= 3636.0ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 46.1ms (= 4609.8ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.27 (= 46.1ms / 36.4ms)

OneFlow resnet50 time: 27.9ms (= 5589.3ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 39.3ms (= 7851.1ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.40 (= 39.3ms / 27.9ms)

OneFlow resnet50 time: 25.1ms (= 5010.9ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 38.4ms (= 7678.1ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.53 (= 38.4ms / 25.1ms)

OneFlow resnet50 time: 24.3ms (= 4853.5ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 36.2ms (= 7246.2ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.49 (= 36.2ms / 24.3ms)

Copy link
Contributor

Copy link
Contributor

Speed stats:
GPU Name: NVIDIA GeForce RTX 3080 Ti 

❌ OneFlow resnet50 time: 43.3ms (= 4333.1ms / 100, input_shape=[16, 3, 224, 224])
PyTorch resnet50 time: 57.1ms (= 5707.6ms / 100, input_shape=[16, 3, 224, 224])
✔️ Relative speed: 1.32 (= 57.1ms / 43.3ms)

OneFlow resnet50 time: 26.6ms (= 2662.8ms / 100, input_shape=[8, 3, 224, 224])
PyTorch resnet50 time: 38.1ms (= 3805.5ms / 100, input_shape=[8, 3, 224, 224])
✔️ Relative speed: 1.43 (= 38.1ms / 26.6ms)

OneFlow resnet50 time: 18.7ms (= 3732.9ms / 200, input_shape=[4, 3, 224, 224])
PyTorch resnet50 time: 36.9ms (= 7373.1ms / 200, input_shape=[4, 3, 224, 224])
✔️ Relative speed: 1.98 (= 36.9ms / 18.7ms)

OneFlow resnet50 time: 17.0ms (= 3397.8ms / 200, input_shape=[2, 3, 224, 224])
PyTorch resnet50 time: 33.9ms (= 6782.8ms / 200, input_shape=[2, 3, 224, 224])
✔️ Relative speed: 2.00 (= 33.9ms / 17.0ms)

OneFlow resnet50 time: 17.2ms (= 3449.2ms / 200, input_shape=[1, 3, 224, 224])
PyTorch resnet50 time: 30.0ms (= 6001.1ms / 200, input_shape=[1, 3, 224, 224])
✔️ Relative speed: 1.74 (= 30.0ms / 17.2ms)

OneFlow swin dataloader time: 0.199s (= 39.825s / 200, num_workers=1)
PyTorch swin dataloader time: 0.128s (= 25.699s / 200, num_workers=1)
Relative speed: 0.645 (= 0.128s / 0.199s)

OneFlow swin dataloader time: 0.055s (= 11.013s / 200, num_workers=4)
PyTorch swin dataloader time: 0.032s (= 6.451s / 200, num_workers=4)
Relative speed: 0.586 (= 0.032s / 0.055s)

OneFlow swin dataloader time: 0.032s (= 6.357s / 200, num_workers=8)
PyTorch swin dataloader time: 0.017s (= 3.311s / 200, num_workers=8)
Relative speed: 0.521 (= 0.017s / 0.032s)

❌ OneFlow resnet50 time: 49.3ms (= 4933.4ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 64.9ms (= 6486.6ms / 100, input_shape=[16, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.31 (= 64.9ms / 49.3ms)

OneFlow resnet50 time: 36.6ms (= 3655.9ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 44.6ms (= 4463.2ms / 100, input_shape=[8, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.22 (= 44.6ms / 36.6ms)

OneFlow resnet50 time: 28.0ms (= 5608.1ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 39.0ms (= 7803.2ms / 200, input_shape=[4, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.39 (= 39.0ms / 28.0ms)

OneFlow resnet50 time: 25.0ms (= 4993.3ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 38.6ms (= 7729.7ms / 200, input_shape=[2, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.55 (= 38.6ms / 25.0ms)

OneFlow resnet50 time: 24.2ms (= 4831.2ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
PyTorch resnet50 time: 35.8ms (= 7167.3ms / 200, input_shape=[1, 3, 224, 224], ddp, world size=2)
✔️ Relative speed: 1.48 (= 35.8ms / 24.2ms)

@lihuizhao lihuizhao merged commit b229019 into master Jan 23, 2024
20 checks passed
@lihuizhao lihuizhao deleted the dev_add_jvp_jacobian_hessian branch January 23, 2024 08:34
Copy link
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants