trainer.test(datamodule=dm) stores reference to wrong checkpoint

[ananyahjha93]

🐛 Bug

When finetuning from saved weights in bolts, trainer.test() picks up reference to checkpoints which have already been deleted or not yet created.
Checkpoint created using default trainer options, no callbacks added from the user's side.

Please reproduce using [the BoringModel and post here]

Not sure how to reproduce fine-tuning from a checkpoint using the boring model.

To Reproduce

1. clone bolts using git clone https://github.com/PyTorchLightning/pytorch-lightning-bolts.git
2. cd to pl_bolts/models/self_supervised/swav/
3. wget 'https://pl-bolts-weights.s3.us-east-2.amazonaws.com/swav/checkpoints/swav_stl10.pth.tar'
4. python swav_finetuner.py --ckpt swav_stl10.pth.tar --dataset stl10 --batch_size 256 --gpus 1 --learning_rate 0.1

Latest saved checkpoint is say 'epoch=33.ckpt' but line 712 in trainer.py looks for other saved checkpoints which might be epochs before or after the one present in checkpoints folder.

  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 712, in test
    results = self.__test_using_best_weights(ckpt_path, test_dataloaders)

Error:

FileNotFoundError: [Errno 2] No such file or directory: '/home/jovyan/pytorch_lightning_bolts/pl_bolts/models/self_supervised/swav/lightning_logs/version_3/checkpoints/epoch=7.ckpt'
FileNotFoundError: [Errno 2] No such file or directory: '/home/jovyan/pytorch_lightning_bolts/pl_bolts/models/self_supervised/swav/lightning_logs/version_3/checkpoints/epoch=21.ckpt'
FileNotFoundError: [Errno 2] No such file or directory: '/home/jovyan/pytorch_lightning_bolts/pl_bolts/models/self_supervised/swav/lightning_logs/version_3/checkpoints/epoch=37.ckpt'

Expected behavior

trainer.test(datamodule=dm) should pickup the reference to the correct checkpoint saved in lightning_logs/version_x/checkpoints

Environment

PyTorch Lightning version 1.0.4+ (tested with both 1.0.4 and 1.0.6)
bolts from master

• PyTorch Version (e.g., 1.0): 1.6
• OS (e.g., Linux): linux
• How you installed PyTorch (conda, pip, source): pip
• Build command you used (if compiling from source):
• Python version: 3.7
• CUDA/cuDNN version:
• GPU models and configuration: V100s
• Any other relevant information:

Additional context

[Borda]

you please say how the checkpoint was created? and what Pl version was used?

[ananyahjha93]

@Borda added