optim.lr_scheduler.CyclicLR (master only: not released) is buggy when not using momentum

[Youssefares]

Issue description

If I use an optimizer like adam with no momentum and follow the message of line 578 below here, passing cycle_momentum=False, line 584 throws a KeyError because key 'momentum' is not set.

pytorch/torch/optim/lr_scheduler.py

Lines 576 to 585 in 173f224

	if cycle_momentum:
	if 'momentum' not in optimizer.defaults:
	raise ValueError('optimizer must support momentum with `cycle_momentum` option enabled')
	base_momentums = self._format_param('base_momentum', optimizer, base_momentum)
	if last_epoch == -1:
	for momentum, group in zip(base_momentums, optimizer.param_groups):
	group['momentum'] = momentum
	self.base_momentums = list(map(lambda group: group['momentum'], optimizer.param_groups))
	self.max_momentums = self._format_param('max_momentum', optimizer, max_momentum)

1. I believe the last two lines should be indented under the if-condition as well.
2. I also think the expected behavior is that cycle_momentum is set to False by default when the passed in optimizer doesn't support momentum, instead of throwing an error.

Code example

optimizer = optim.Adam(params_to_update, lr=5e-4)
scheduler = CyclicLR(optimizer, base_lr=5e-6, max_lr=5e-2, cycle_momentum=False, step_size_up=2500)

System Info

In my setup, I am using the stable version of pytorch, but I copied over CyclicLR and relative imports from master and I am using it in my project.

Collecting environment information...
PyTorch version: 1.0.1.post2
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Debian GNU/Linux 9.8 (stretch)
GCC version: (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
CMake version: Could not collect

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: Tesla P4
Nvidia driver version: 410.72
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] intel-numpy==1.15.1
[pip3] numpy==1.15.1
[pip3] torch==1.0.1.post2
[pip3] torchvision==0.2.2.post3
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.1                      144
[conda] mkl-service               1.1.2            py37he904b0f_5
[conda] mkl_fft                   1.0.6            py37hd81dba3_0
[conda] mkl_random                1.0.2            py37hd81dba3_0
[conda] pytorch                   1.0.1           py3.7_cuda10.0.130_cudnn7.4.2_2    pytorch
[conda] torchvision               0.2.2                      py_3    pytorch

[Youssefares]

Let me know if you'd like me to work on this 👨‍💻

[soumith]

@Youssefares sure thing, go for it. Thank you!

[trias702]

This bug is still present in the official PyTorch 1.1.0 which was released Apr 30, 2019. It's a really bad bug since CyclicLR is a big feature of the new pytorch and this bug causes CyclicLR to fail on any optimizer which doesn't support momentum. It's a fairly easy fix too, just indent two lines of code.

[BorisMarjanovic]

When will this bug be fixed?

[memahesh]

This bug is still present. Is anyone working to fix this ? If not, maybe I can work on it.

[memahesh]

@soumith and @Youssefares
Can any of you tell me if its fixed or not ?

[Youssefares]

@memahesh
I haven't had time to fix this I am afraid.
Sorry for any delay I might have caused. I think a pull request is already open for it though. So you can follow the progress there.

[jeroenvuurens]

I think you are correct, if you indent the second and third to last lines in the init it works.

[gchanan]

Fixed in #20401.

[meghbhalerao]

Hmm. Is anyone getting this error in pytorch 1.6.0?

[ghost]

I use PyTorch 1.4, but I still get the following error when using Adam without momentum:
ValueError: optimizer must support momentum with cycle_momentum option enabled

What I do now instead is to use SGD with momentum ...

[meghbhalerao]

Oh, okay. So basically there are some compatibility issues with the optimizer and lr scheduler for certain combinations of the two?

[ghost]

Exactly. At least it seems so.

I hope this will be fixed at some point. :-)

[kaushal-py]

The bug exists in pytorch 1.6 as well.

[snk4tr]

The bug exists in PyTorch 1.7.0 😞

[Jungjee]

Is there anyone who knows why this issue is closed although this bug exists in the latest stable release(1.7.0)??

Or is there a solution for this phenomenon except for setting cycle_momentum=False

[Moldoteck]

Any updates for this?

[thomashirtz]

Is someone currently working on this ? (This issue is not solved)

[patricio-astudillo]

The bug exists in PyTorch 1.9.0

[cowwoc]

@gchanan Please reopen this issue. It is not fixed.

[thomashirtz]

Does someone know/can explain me how to fix it ? I can try to make a PR

[gonzrubio]

The bug exists in PyTorch 2.0.0