lr_find doesn't return the correct suggestion if some losses are nan

[binshengliu]

🐛 Bug

lr_finder doesn't return the correct suggestion if some losses are nan. The returned loss is the one corresponding to the nan value, which is very big in my case.

To Reproduce

This depends on the dataset. Please see the code sample.

Code sample

I believe this is caused by numpy. The related code should be https://github.com/PyTorchLightning/pytorch-lightning/blob/b84b02400a312240a6429c186cc63514eeb45a82/pytorch_lightning/trainer/lr_finder.py#L325

example_losses = [0.90, 0.89, 0.87, 0.86, 0.85, 0.84]
print(np.gradient(example_losses).argmin())
example_losses = [0.90, 0.89, 0.87, 0.86, 0.85, 0.84, float('nan')]
print(np.gradient(example_losses).argmin())

Output:

1
5

Expected behavior

Return the correct suggested loss.

Environment

• CUDA:
  • GPU:
  • available: False
  • version: 10.2
• Packages:
  • numpy: 1.18.4
  • pyTorch_debug: False
  • pyTorch_version: 1.5.0
  • pytorch-lightning: 0.7.6
  • tensorboard: 2.2.0
  • tqdm: 4.45.0
• System:
  • OS: Linux
  • architecture:
    • 64bit
  • processor:
  • python: 3.7.6
  • version: Proposal for help #1 SMP Debian 4.19.118-2 (2020-04-29)

Additional context

NA

[github-actions]

Hi! thanks for your contribution!, great first issue!

[SkafteNicki]

Good catch. I guess that we can not do much about how np.gradient behave but we can filter Nan values before doing the calculation. @binshengliu are you up for doing a PR?

[binshengliu]

Filtering out nan would be a reasonable approach. Locally I just reset nan to inf also to avoid dealing with indexes.

Sorry I'm quite inundated with my own projects recently and may not have enough time to shape a proper PR.