Add the `maximize` flag to all optimizers #68052

albanD · 2021-11-09T08:47:55Z

OliverFM · 2021-11-10T08:00:32Z

Which one is the next highest priority? Adam?
I would like to try do another at some point in the next week or two

albanD · 2021-11-10T09:47:48Z

Adam sounds like a good candidate. I re-ordered the list above roughly to be in order.

wukong1992 · 2021-11-11T03:52:18Z

Hey!
I am new comer, and want to contributing to torch. And which optim else need to be done，i will spend some time for the PR。

OliverFM · 2021-11-11T07:45:23Z

Hey! I am new comer, and want to contributing to torch. And which optim else need to be done，i will spend some time for the PR。

Hey @wukong1992! Welcome!
I am also new here, however I implemented the fix for SGD, and opened #68164 for Adam.

In my opinion, the best first step is to setup the torch repo locally and run tests. After that, you'd probably want to pick one of the optimizers and mention here that you are working on that implementation.
You can look at the existing implementation in SGD for reference on how this ought to be done and test_sgd in torch/test_optim.py for an example of the testing that you'd need to do.

I hope this helps!

wukong1992 · 2021-11-11T11:26:01Z

Hey! I am new comer, and want to contributing to torch. And which optim else need to be done，i will spend some time for the PR。

Hey @wukong1992! Welcome! I am also new here, however I implemented the fix for SGD, and opened #68164 for Adam.

In my opinion, the best first step is to setup the torch repo locally and run tests. After that, you'd probably want to pick one of the optimizers and mention here that you are working on that implementation. You can look at the existing implementation in SGD for reference on how this ought to be done and test_sgd in torch/test_optim.py for an example of the testing that you'd need to do.

I hope this helps!

OK！Thank you so much！

Summary: A new attribute has been added to SGD and will be added to other optimizers in the future. We need to make a corresponding change to `OptimizerConfig` pytorch/pytorch#68052 Differential Revision: D32513683 fbshipit-source-id: addba19d29dd04792ce770ca4df82388146668ad

Summary: Pull Request resolved: facebookresearch#581 A new attribute has been added to SGD and will be added to other optimizers in the future. We need to make a corresponding change to `OptimizerConfig` pytorch/pytorch#68052 Differential Revision: D32513683 fbshipit-source-id: f804db019fb97818359dd02b50caf8e372e32d66

Summary: Pull Request resolved: facebookresearch#581 A new attribute has been added to SGD and will be added to other optimizers in the future. We need to make a corresponding change to `OptimizerConfig` pytorch/pytorch#68052 Differential Revision: D32513683 fbshipit-source-id: 742f0174bd996a6e3473170a572650124963d50a

Summary: Pull Request resolved: #581 A new attribute has been added to SGD and will be added to other optimizers in the future. We need to make a corresponding change to `OptimizerConfig` pytorch/pytorch#68052 Reviewed By: czxttkl Differential Revision: D32513683 fbshipit-source-id: 61f4042c10f9843f73d886b9d8c1d90baa52c5c1

Summary: Solves the next most important use case in #68052. I have kept the style as close to that in SGD as seemed reasonable, given the slight differences in their internal implementations. All feedback welcome! cc pietern mrshenli pritamdamania87 zhaojuanmao satgera rohan-varma gqchen aazzolini osalpekar jiayisuse SciPioneer H-Huang Pull Request resolved: #68164 Reviewed By: VitalyFedyunin Differential Revision: D32994129 Pulled By: albanD fbshipit-source-id: 65c57c3f3dbbd3e3e5338d51def54482503e8850

OliverFM · 2021-12-17T15:25:05Z

Adam is done, #68164, I may start work on AdamW this week.

albanD · 2021-12-17T16:33:06Z

Thanks!

Adnios · 2021-12-18T01:26:53Z

oops. @OliverFM Have you started AdamW? I just finished AdamW(#70146).

OliverFM · 2021-12-18T07:00:30Z

oops. @OliverFM Have you started AdamW? I just finished AdamW(#70146).

Fortunately not :)
I'll pick up ASGD instead!

Summary: Related issue: #68052 cc pietern mrshenli pritamdamania87 zhaojuanmao satgera rohan-varma gqchen aazzolini osalpekar jiayisuse SciPioneer H-Huang Pull Request resolved: #70146 Reviewed By: malfet Differential Revision: D33254561 Pulled By: albanD fbshipit-source-id: f190c836a4162f936c5953e076747c345df21421

Added the maximize flag to Adadelta optimizer (#68052) and adjusted tests to take maximize into account. Pull Request resolved: #75330 Approved by: https://github.com/cpuhrsch

Summary: Added the maximize flag to Adadelta optimizer (#68052) and adjusted tests to take maximize into account. Pull Request resolved: #75330 Approved by: https://github.com/cpuhrsch Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/58a44523c1f74093ad9a05513a32c45a109b6c55 Reviewed By: b0noI Differential Revision: D35510886 fbshipit-source-id: 3d123365c381a83a3b2f71eb6a889641d96af766

zaxtax · 2022-04-14T12:24:14Z

I'll handle the rest of these!

This adds maximize to Adagrad (#68052) along with updates the respective tests. Pull Request resolved: #75968 Approved by: https://github.com/albanD

This adds maximize to Adagrad (#68052) along with updates the respective tests. Pull Request resolved: #75968 Approved by: https://github.com/albanD (cherry picked from commit 6642e88)

Summary: This adds maximize to Adagrad (#68052) along with updates the respective tests. Pull Request resolved: #75968 Approved by: https://github.com/albanD Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/6642e88ad23bfcf1349f2bce06f3490e49f195bf Reviewed By: seemethere Differential Revision: D35785877 fbshipit-source-id: d494b629f3a1b77cec3b3f1ab507516ddea0005a

Added the maximize flag #68052 to Adamax optimizer and updates the respective tests. Pull Request resolved: #77409 Approved by: https://github.com/albanD

Summary: Added the maximize flag #68052 to Adamax optimizer and updates the respective tests. Pull Request resolved: #77409 Approved by: https://github.com/albanD Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/2a496e2f80a9937c336aaabf13f99cbc3983998c Reviewed By: atalman Differential Revision: D36421575 fbshipit-source-id: 298aa514ffa138c01daa8c538ba757b2af31b2e1

Added the maximize flag #68052 to ASGD optimizer and updates the respective tests. Pull Request resolved: #80323 Approved by: https://github.com/albanD

Added the maximize flag #68052 to RMSprop optimizer and updates the respective tests. Pull Request resolved: #80326 Approved by: https://github.com/albanD

Added the maximize flag #68052 to rprop optimizer and updates the respective tests. Pull Request resolved: #80335 Approved by: https://github.com/albanD

Added the maximize flag #68052 to SparseAdam optimizer and updates the respective tests. Pull Request resolved: #80336 Approved by: https://github.com/albanD

Summary: Added the maximize flag #68052 to RMSprop optimizer and updates the respective tests. Pull Request resolved: #80326 Approved by: https://github.com/albanD Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/a1fd5b42730924c5e5a2beb5dabc779725768830 Reviewed By: mehtanirav Differential Revision: D37717279 Pulled By: mehtanirav fbshipit-source-id: a87ef2779dd8ec28513c7cff61b7cbbbf675b913

Summary: Added the maximize flag #68052 to SparseAdam optimizer and updates the respective tests. Pull Request resolved: #80336 Approved by: https://github.com/albanD Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/f24c94d7ae00a7acc1fc3d543e6217509cfeb885 Reviewed By: mehtanirav Differential Revision: D37717288 Pulled By: mehtanirav fbshipit-source-id: 89d20b886febdad620b0c7e52592931500a8beef

Added the maximize flag #68052 to ASGD optimizer and updates the respective tests. Pull Request resolved: #81875 Approved by: https://github.com/albanD

Summary: Added the maximize flag #68052 to ASGD optimizer and updates the respective tests. Pull Request resolved: #81875 Approved by: https://github.com/albanD Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/50c655d5e33eb2ec4d501fe3b03647720a3e67c8 Reviewed By: osalpekar Differential Revision: D38119385 Pulled By: osalpekar fbshipit-source-id: cb1a91a94a642b39113ad5a163c7901e64fd80a8

Added the maximize flag #68052 to rprop optimizer and updates the respective tests. Pull Request resolved: #81864 Approved by: https://github.com/albanD

Summary: Added the maximize flag #68052 to rprop optimizer and updates the respective tests. Pull Request resolved: #81864 Approved by: https://github.com/albanD Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/ff75562cffb54d7500a94a1091e06dc9b5c284fc Reviewed By: atalman Differential Revision: D38745528 fbshipit-source-id: 3b59bf641ff08d37921f231cd4a04244f8a045b6

albanD · 2022-10-05T23:51:44Z

We will not do lbfgs as part of this issue as it is more complex than others.

albanD added feature A request for a proper, new feature. module: optimizer Related to torch.optim triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Nov 9, 2021

albanD mentioned this issue Nov 9, 2021

Allow negative learning rates #46480

Closed

OliverFM mentioned this issue Nov 11, 2021

Adds a maximize flag to Adam #68164

Closed

alexnikulkov mentioned this issue Nov 17, 2021

add optimize=False to reagent optimizer configs facebookresearch/ReAgent#581

Closed

crcrpar mentioned this issue Nov 23, 2021

Foreach Functions Tracking Issue #58833

Open

28 tasks

Adnios mentioned this issue Dec 18, 2021

Add the maximize flag to AdamW #70146

Closed

francescocastelli mentioned this issue Apr 6, 2022

Add maximize flag to Adadelta #75330

Closed

pytorchmergebot pushed a commit that referenced this issue Apr 8, 2022

Add maximize flag to Adadelta

58a4452

Added the maximize flag to Adadelta optimizer (#68052) and adjusted tests to take maximize into account. Pull Request resolved: #75330 Approved by: https://github.com/cpuhrsch

zaxtax mentioned this issue Apr 18, 2022

Adding maximize flag to Adagrad #75968

Closed

pytorchmergebot pushed a commit that referenced this issue Apr 20, 2022

Adding maximize flag to Adagrad

6642e88

This adds maximize to Adagrad (#68052) along with updates the respective tests. Pull Request resolved: #75968 Approved by: https://github.com/albanD

malfet pushed a commit that referenced this issue Apr 20, 2022

Adding maximize flag to Adagrad

5097cdc

This adds maximize to Adagrad (#68052) along with updates the respective tests. Pull Request resolved: #75968 Approved by: https://github.com/albanD (cherry picked from commit 6642e88)

zaxtax mentioned this issue May 13, 2022

Adding maximize to Adamax #77409

Closed

pytorchmergebot pushed a commit that referenced this issue May 16, 2022

Adding maximize to Adamax (#77409)

2a496e2

Added the maximize flag #68052 to Adamax optimizer and updates the respective tests. Pull Request resolved: #77409 Approved by: https://github.com/albanD

This was referenced Jun 27, 2022

Adding maximize to ASGD #80323

Closed

Adding maximize to RMSprop #80326

Closed

Adding maximize to rprop #80335

Closed

Adding maximize to SparseAdam #80336

Closed

pytorchmergebot pushed a commit that referenced this issue Jul 8, 2022

Adding maximize to ASGD (#80323)

14bd5bd

Added the maximize flag #68052 to ASGD optimizer and updates the respective tests. Pull Request resolved: #80323 Approved by: https://github.com/albanD

pytorchmergebot pushed a commit that referenced this issue Jul 8, 2022

Adding maximize to RMSprop (#80326)

a1fd5b4

Added the maximize flag #68052 to RMSprop optimizer and updates the respective tests. Pull Request resolved: #80326 Approved by: https://github.com/albanD

pytorchmergebot pushed a commit that referenced this issue Jul 8, 2022

Adding maximize to rprop (#80335)

495aa9b

Added the maximize flag #68052 to rprop optimizer and updates the respective tests. Pull Request resolved: #80335 Approved by: https://github.com/albanD

zaxtax mentioned this issue Jul 8, 2022

Adding maximize to LBFGS #81098

Closed

pytorchmergebot pushed a commit that referenced this issue Jul 8, 2022

Adding maximize to SparseAdam (#80336)

f24c94d

Added the maximize flag #68052 to SparseAdam optimizer and updates the respective tests. Pull Request resolved: #80336 Approved by: https://github.com/albanD

latorrefabian mentioned this issue Jul 18, 2022

Reconsider allowing negative learning rates in torch.optim.SGD (and others) #81633

Closed

This was referenced Jul 21, 2022

Adding maximize to rprop #81864

Closed

Adding maximize to ASGD #81875

Closed

pytorchmergebot pushed a commit that referenced this issue Jul 22, 2022

Adding maximize to ASGD (#81875)

50c655d

Added the maximize flag #68052 to ASGD optimizer and updates the respective tests. Pull Request resolved: #81875 Approved by: https://github.com/albanD

pytorchmergebot pushed a commit that referenced this issue Aug 16, 2022

Adding maximize to rprop (#81864)

ff75562

Added the maximize flag #68052 to rprop optimizer and updates the respective tests. Pull Request resolved: #81864 Approved by: https://github.com/albanD

juan43ramirez mentioned this issue Aug 26, 2022

Do maximize=True for dual_optimizers cooper-org/cooper#49

Open

albanD mentioned this issue Oct 5, 2022

Add maximize support to LBFGS optimizer #86339

Open

albanD closed this as completed Oct 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the `maximize` flag to all optimizers #68052

Add the `maximize` flag to all optimizers #68052

albanD commented Nov 9, 2021 •

edited

OliverFM commented Nov 10, 2021

albanD commented Nov 10, 2021

wukong1992 commented Nov 11, 2021

OliverFM commented Nov 11, 2021

wukong1992 commented Nov 11, 2021

OliverFM commented Dec 17, 2021

albanD commented Dec 17, 2021

Adnios commented Dec 18, 2021 •

edited

OliverFM commented Dec 18, 2021

zaxtax commented Apr 14, 2022

albanD commented Oct 5, 2022

Add the maximize flag to all optimizers #68052

Add the maximize flag to all optimizers #68052

Comments

albanD commented Nov 9, 2021 • edited

OliverFM commented Nov 10, 2021

albanD commented Nov 10, 2021

wukong1992 commented Nov 11, 2021

OliverFM commented Nov 11, 2021

wukong1992 commented Nov 11, 2021

OliverFM commented Dec 17, 2021

albanD commented Dec 17, 2021

Adnios commented Dec 18, 2021 • edited

OliverFM commented Dec 18, 2021

zaxtax commented Apr 14, 2022

albanD commented Oct 5, 2022

Add the `maximize` flag to all optimizers #68052

Add the `maximize` flag to all optimizers #68052

albanD commented Nov 9, 2021 •

edited

Adnios commented Dec 18, 2021 •

edited