Parity tests for functional optimizer step_param #61756

rohan-varma · 2021-07-16T03:54:16Z

Stack from ghstack:

DDP will support running optimizer as communication hook with
optimizers that support a per-parameter/gradient step function step_param.
Add parity tests as we implement more optimizers that support step_param to
ensure parity with regular optimizers.

Differential Revision: D29727549

NOTE FOR REVIEWERS: This PR has internal Facebook specific changes or comments, please review them on Phabricator!

DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)! [ghstack-poisoned]

facebook-github-bot · 2021-07-16T03:54:22Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/61756
📄 Preview docs built from this PR

💊 CI failures summary and remediations

As of commit c6c9554 (more details on the Dr. CI page):

1/1 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

Linux CI (pytorch-linux-bionic-py3.8-gcc9-coverage) / render_test_results (default) (1/1)

Step: "Checkout PyTorch" (full log | diagnosis details | 🔁 rerun)

rm: cannot remove '/home/ec2-user/actions-runne..._pipelines/verify-pipeline.yml': Permission denied

rm: cannot remove '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.azure_pipelines/job_templates/prepare-build-template.yml': Permission denied
rm: cannot remove '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.azure_pipelines/job_templates/pytorch-template-unix.yml': Permission denied
rm: cannot remove '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.azure_pipelines/job_templates/pytorch-template-win.yml': Permission denied
rm: cannot remove '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.azure_pipelines/job_templates/set-environment-variables.yml': Permission denied
rm: cannot remove '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.azure_pipelines/job_templates/wheel-wait-job-template.yml': Permission denied
rm: cannot remove '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.azure_pipelines/job_templates/wheel-wait-template.yml': Permission denied
rm: cannot remove '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.azure_pipelines/build-pipeline.yml': Permission denied
rm: cannot remove '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.azure_pipelines/daily-pipeline.yml': Permission denied
rm: cannot remove '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.azure_pipelines/nightly-pytorch-tests-pipeline.yml': Permission denied
rm: cannot remove '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.azure_pipelines/pytorch-tests-pipeline.yml': Permission denied
rm: cannot remove '/home/ec2-user/actions-runner/_work/pytorch/pytorch/.azure_pipelines/verify-pipeline.yml': Permission denied

2021-07-26T21:45:45.7593602Z Post job cleanup.
2021-07-26T21:45:45.8384744Z Cleaning up orphan processes

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)! [ghstack-poisoned]

Pull Request resolved: #61756 DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. ghstack-source-id: 133659711 Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)!

DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)! [ghstack-poisoned]

Pull Request resolved: #61756 DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. ghstack-source-id: 133682132 Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)!

DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)! [ghstack-poisoned]

Pull Request resolved: #61756 DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. ghstack-source-id: 133919961 Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)!

wayi1 · 2021-07-20T23:42:30Z

test/test_functional_optim.py

+            # Functional optimizer step_param
+            for param in module_functional.parameters():
+                grad = param.grad
+                optim_functional.step_param(param, grad)


It seems that the first arg of step_param should be a tensor instead of nn.parameter.Parameter. Is there an implicit type casting here?

Good point! According to these docs: https://github.com/pytorch/pytorch/blob/master/torch/nn/parameter.py#L7 a Parameter is a kind of tensor (and also subclasses from Tensor), so I think step_param should work regardless of whether it is passed a parameter or tensor.

DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)! [ghstack-poisoned]

Pull Request resolved: #61756 DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. ghstack-source-id: 133945706 Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)!

DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)! [ghstack-poisoned]

Pull Request resolved: #61756 DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. ghstack-source-id: 134052041 Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)!

DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)! [ghstack-poisoned]

Pull Request resolved: #61756 DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. ghstack-source-id: 134272866 Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)!

DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)! [ghstack-poisoned]

facebook-github-bot · 2021-07-27T02:05:19Z

This pull request has been merged in 69adb21.

Pull Request resolved: pytorch/pytorch#61756 DDP will support running optimizer as communication hook with optimizers that support a per-parameter/gradient step function `step_param`. Add parity tests as we implement more optimizers that support step_param to ensure parity with regular optimizers. ghstack-source-id: 133985151 Differential Revision: [D29727549](https://our.internmc.facebook.com/intern/diff/D29727549/) **NOTE FOR REVIEWERS**: This PR has internal Facebook specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D29727549/)!

facebook-github-bot added the cla signed label Jul 16, 2021

This was referenced Jul 16, 2021

[DDP] Add API to get model parameters in hook #61637

Closed

[DDP] Implement a hook which performs FunctionalSGD step. #61678

Closed

rohan-varma requested review from wayi1 and zhaojuanmao July 20, 2021 20:43

wayi1 reviewed Jul 20, 2021

View reviewed changes

wayi1 approved these changes Jul 20, 2021

View reviewed changes

rohan-varma mentioned this pull request Jul 21, 2021

[WIP, not for review] Remove copy_bucket_to_grad when optimizer runs as hook #61961

Closed

This was referenced Jul 23, 2021

[Functional Optim] Test kwargs parity for SGD #62078

Closed

[DDP/Functional Optim] Support kwarg arguments #62079

Closed

rohan-varma added 3 commits July 22, 2021 23:50

rohan-varma added 2 commits July 23, 2021 19:08

rohan-varma mentioned this pull request Jul 26, 2021

[Reland] [DDP] Implement a hook which performs FunctionalSGD step. #62177

Closed

facebook-github-bot closed this in 69adb21 Jul 27, 2021

facebook-github-bot added the Merged label Jul 27, 2021

facebook-github-bot deleted the gh/rohan-varma/355/head branch July 30, 2021 14:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parity tests for functional optimizer step_param #61756

Parity tests for functional optimizer step_param #61756

rohan-varma commented Jul 16, 2021 •

edited

facebook-github-bot commented Jul 16, 2021 •

edited

wayi1 Jul 20, 2021

rohan-varma Jul 20, 2021

facebook-github-bot commented Jul 27, 2021

Parity tests for functional optimizer step_param #61756

Parity tests for functional optimizer step_param #61756

Conversation

rohan-varma commented Jul 16, 2021 • edited

facebook-github-bot commented Jul 16, 2021 • edited

🔗 Helpful links

💊 CI failures summary and remediations

🕵️ 1 new failure recognized by patterns

Linux CI (pytorch-linux-bionic-py3.8-gcc9-coverage) / render_test_results (default) (1/1)

wayi1 Jul 20, 2021

Choose a reason for hiding this comment

rohan-varma Jul 20, 2021

Choose a reason for hiding this comment

facebook-github-bot commented Jul 27, 2021

rohan-varma commented Jul 16, 2021 •

edited

facebook-github-bot commented Jul 16, 2021 •

edited