Incompatible Trainer.test()

[isvoboda]

🐛 Bug

Trainer.test() stops on missing definitions of train & validation dataloaders methods.

To Reproduce

Steps to reproduce the behavior:

• Define LightningModule without train & validation dataloaders
• Train such model with data loaders provided to Trainer.fit(...) method
• Load model from checkpoint and test it with Trainer.test(...) method

test failes on :

'No `train_dataloader()` method defined. Lightning `Trainer` expects as minimum a'
pytorch_lightning.utilities.exceptions.MisconfigurationException: No `train_dataloader()` method defined. Lightning `Trainer` expects as minimum a `training_step()`, `training_dataloader()` and `configure_optimizers()` to be defined.

Code sample

test_data = data_loader(...)
estimator = MyEstimator.load_from_checkpoint(checkpoint_path=CKPT_PATH)
trainer = pl.Trainer(...)
trainer.test(model=estimator, test_dataloaders=test_data)

Expected behaviour

Run the test with provided test data loader, and do not stop on condition related to missing train & validation dataloaders which actually are not mandatory.

Environment

* Packages:
        - numpy:             1.18.4
        - pyTorch_debug:     False
        - pyTorch_version:   1.5.0
        - pytorch-lightning: 0.7.5
        - tensorboard:       2.2.1
        - tqdm:              4.46.0
* System:
        - OS:                Linux
        - architecture:
                - 64bit
                - ELF
        - processor:         
        - python:            3.8.2
        - version:           #1 SMP Wed, 29 Apr 2020 16:23:03 +0000

[github-actions]

Hi! thanks for your contribution!, great first issue!

[isvoboda]

This issue relates to #1195, fit and test should be decoupled. Is there any consensus or proposition of further steps?

[rohitgr7]

Can't we just add

if not self.testing:
    self.check_model_configuration(model)

here because during testing the testing configuraton is already checked here.

[iakremnev]

Also, using trainer.test(model) on loaded checkpoint doesn't work without explicitly passing test_dataloader parameter even if all *_dataloader methods are defined. Example.
This use case isn't documented in the 0.7.5 documentation but evidently was relevant at some point: https://pytorch-lightning.readthedocs.io/en/latest/test_set.html#test-pre-trained-model.
I suggest to either:

• fix Test set documentation
• (preferred) allow to call trainer.test(model) without specifying test_dataloader parameter. After all, what's the point of defining Model.test_dataloader method?

[rohitgr7]

@iakremnev can you fix the link? It's asking for request access.

[iakremnev]

@rohitgr7 oops, fixed it.

[rohitgr7]

Error coming from this line. Why is it checking whether test_dataloader() returns something or not. If we just check whether it is overridden or not wouldn't that be enough just like it is done in case of .fit() here.

[rohitgr7]

I would suggest

gave_test_loader = isinstance(model.test_dataloader, _PatchDataLoader) or self.is_overridden('test_dataloader', model)

here, and

if not self.testing:
    self.check_model_configuration(model)

here

[Borda]

it is the same as #1754