Skip to content

Commit 325852c

Browse files
enabled no returns from eval (Lightning-AI#2446)
* enabled no returns from eval * fixed docs * fixed docs * fixed docs * fixed docs * fixed docs * fixed docs * fixed docs * fixed docs * fixed docs * fixed docs * fixed docs * fixed docs
1 parent fa2233f commit 325852c

File tree

8 files changed

+153
-35
lines changed

8 files changed

+153
-35
lines changed

docs/source/bolts.rst

+89
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
Bolts
2+
=====
3+
`PyTorch Lightning Bolts <https://pytorch-lightning-bolts.readthedocs.io/en/latest/>`_, is our official collection
4+
of prebuilt models across many research domains.
5+
6+
.. code-block:: bash
7+
8+
pip install pytorch-lightning-bolts
9+
10+
In bolts we have:
11+
12+
- A collection of pretrained state-of-the-art models.
13+
- A collection of models designed to bootstrap your research.
14+
- A collection of Callbacks, transforms, full datasets.
15+
- All models work on CPUs, TPUs, GPUs and 16-bit precision.
16+
17+
-----------------
18+
19+
Quality control
20+
---------------
21+
Bolts are built-by the Lightning community and contributed to bolts.
22+
The lightning team guarantees that contributions are:
23+
24+
- Rigorously Tested (CPUs, GPUs, TPUs)
25+
- Rigorously Documented
26+
- Standardized via PyTorch Lightning
27+
- Optimized for speed
28+
- Checked for correctness
29+
30+
---------
31+
32+
Example 1: Pretrained, prebuilt models
33+
--------------------------------------
34+
35+
.. code-block:: python
36+
37+
from pl_bolts.models import VAE, GPT2, ImageGPT, PixelCNN
38+
from pl_bolts.models.self_supervised import AMDIM, CPCV2, SimCLR, MocoV2
39+
from pl_bolts.models import LinearRegression, LogisticRegression
40+
from pl_bolts.models.gans import GAN
41+
from pl_bolts.callbacks import PrintTableMetricsCallback
42+
from pl_bolts.datamodules import FashionMNISTDataModule, CIFAR10DataModule, ImagenetDataModule
43+
44+
------------
45+
46+
Example 2: Extend for faster research
47+
-------------------------------------
48+
Bolts are contributed with benchmarks and continuous-integration tests. This means
49+
you can trust the implementations and use them to bootstrap your resarch much faster.
50+
51+
.. code-block:: python
52+
53+
from pl_bolts.models import ImageGPT
54+
from pl_bolts.self_supervised import SimCLR
55+
56+
class VideoGPT(ImageGPT):
57+
58+
def training_step(self, batch, batch_idx):
59+
x, y = batch
60+
x = _shape_input(x)
61+
62+
logits = self.gpt(x)
63+
simclr_features = self.simclr(x)
64+
65+
# -----------------
66+
# do something new with GPT logits + simclr_features
67+
# -----------------
68+
69+
loss = self.criterion(logits.view(-1, logits.size(-1)), x.view(-1).long())
70+
71+
logs = {"loss": loss}
72+
return {"loss": loss, "log": logs}
73+
74+
----------
75+
76+
Example 3: Callbacks
77+
--------------------
78+
We also have a collection of callbacks.
79+
80+
.. code-block:: python
81+
82+
from pl_bolts.callbacks import PrintTableMetricsCallback
83+
import pytorch_lightning as pl
84+
85+
trainer = pl.Trainer(callbacks=[PrintTableMetricsCallback()])
86+
87+
# loss│train_loss│val_loss│epoch
88+
# ──────────────────────────────
89+
# 2.2541470527648926│2.2541470527648926│2.2158432006835938│0

docs/source/callbacks.rst

+8-8
Original file line numberDiff line numberDiff line change
@@ -49,14 +49,14 @@ We successfully extended functionality without polluting our super clean
4949
----------------
5050

5151
Best Practices
52-
==============
53-
54-
1. Callbacks should be isolated in their functionality. Your callback should not rely on the
55-
behavior of other callbacks in order to work properly.
56-
2. Do not manually call methods from the callback. The callbacks are designed to be
57-
invoked at specific times during training. Directly calling methods (eg. `on_validation_end`)
58-
is strongly discouraged.
59-
3. Whenever possible, your callbacks should not depend on the order in which they are executed.
52+
--------------
53+
The following are best practices when using/designing callbacks.
54+
55+
1. Callbacks should be isolated in their functionality.
56+
2. Your callback should not rely on the behavior of other callbacks in order to work properly.
57+
3. Do not manually call methods from the callback.
58+
4. Directly calling methods (eg. `on_validation_end`) is strongly discouraged.
59+
5. Whenever possible, your callbacks should not depend on the order in which they are executed.
6060

6161

6262
---------

docs/source/conf.py

+1
Original file line numberDiff line numberDiff line change
@@ -139,6 +139,7 @@
139139
'api/pytorch_lightning.rst',
140140
'api/pl_examples.*',
141141
'api/modules.rst',
142+
'PULL_REQUEST_TEMPLATE.md',
142143

143144
# deprecated/renamed:
144145
'api/pytorch_lightning.logging.*', # TODO: remove in v0.9.0

docs/source/hooks.rst

+4
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ Hooks lifecycle
2020
Training set-up
2121
^^^^^^^^^^^^^^^
2222

23+
- :meth:`~pytorch_lightning.core.lightning.LightningModule.prepare_data`
24+
- :meth:`~pytorch_lightning.core.lightning.LightningModule.setup`
2325
- :meth:`~pytorch_lightning.core.lightning.LightningModule.init_ddp_connection`
2426
- :meth:`~pytorch_lightning.trainer.optimizers.TrainerOptimizersMixin.init_optimizers`
2527
- :meth:`~pytorch_lightning.core.lightning.LightningModule.configure_apex`
@@ -30,6 +32,8 @@ Training set-up
3032
- :meth:`~pytorch_lightning.core.lightning.LightningModule.summarize`
3133
- :meth:`~pytorch_lightning.trainer.training_io.TrainerIOMixin.restore_weights`
3234

35+
.. warning:: `prepare_data` is only called from global_rank=0. Don't assign state (self.something), use `setup` for that
36+
3337
----------
3438

3539
Training loop

docs/source/index.rst

+7-2
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,13 @@ PyTorch Lightning Documentation
2727
hooks
2828
trainer
2929

30+
.. toctree::
31+
:maxdepth: 1
32+
:name: Bolts
33+
:caption: Bolts
34+
35+
bolts
36+
3037
.. toctree::
3138
:maxdepth: 1
3239
:name: Community Examples
@@ -35,7 +42,6 @@ PyTorch Lightning Documentation
3542
Contextual Emotion Detection (DoubleDistilBert) <https://github.com/PyTorchLightning/emotion_transformer>
3643
Cotatron: Transcription-Guided Speech Encoder <https://github.com/mindslab-ai/cotatron>
3744
FasterRCNN object detection + Hydra <https://github.com/PyTorchLightning/wheat>
38-
Generative Adversarial Network <https://colab.research.google.com/drive/1F_RNcHzTfFuQf-LeKvSlud6x7jXYkG31#scrollTo=TyYOdg8g77P0>
3945
Hyperparameter optimization with Optuna <https://github.com/optuna/optuna/blob/master/examples/pytorch_lightning_simple.py>
4046
Image Inpainting using Partial Convolutions <https://github.com/ryanwongsa/Image-Inpainting>
4147
MNIST on TPU <https://colab.research.google.com/drive/1-_LKx4HwAxl5M6xPJmqAAu444LTDQoa3#scrollTo=BHBz1_AnamN_>
@@ -100,7 +106,6 @@ PyTorch Lightning Documentation
100106
CODE_OF_CONDUCT.md
101107
CONTRIBUTING.md
102108
BECOMING_A_CORE_CONTRIBUTOR.md
103-
PULL_REQUEST_TEMPLATE.md
104109
governance.md
105110

106111
Indices and tables

pytorch_lightning/core/lightning.py

+19-2
Original file line numberDiff line numberDiff line change
@@ -1337,11 +1337,19 @@ def train_dataloader(self) -> DataLoader:
13371337
The dataloader you return will not be called every epoch unless you set
13381338
:paramref:`~pytorch_lightning.trainer.Trainer.reload_dataloaders_every_epoch` to ``True``.
13391339
1340-
It's recommended that all data downloads and preparation happen in :meth:`prepare_data`.
1340+
For data processing use the following pattern:
1341+
1342+
- download in :meth:`prepare_data`
1343+
- process and split in :meth:`setup`
1344+
1345+
However, the above are only necessary for distributed processing.
1346+
1347+
.. warning:: do not assign state in prepare_data
13411348
13421349
- :meth:`~pytorch_lightning.trainer.Trainer.fit`
13431350
- ...
13441351
- :meth:`prepare_data`
1352+
- :meth:`setup`
13451353
- :meth:`train_dataloader`
13461354
13471355
Note:
@@ -1383,11 +1391,20 @@ def test_dataloader(self) -> Union[DataLoader, List[DataLoader]]:
13831391
The dataloader you return will not be called every epoch unless you set
13841392
:paramref:`~pytorch_lightning.trainer.Trainer.reload_dataloaders_every_epoch` to ``True``.
13851393
1386-
It's recommended that all data downloads and preparation happen in :meth:`prepare_data`.
1394+
For data processing use the following pattern:
1395+
1396+
- download in :meth:`prepare_data`
1397+
- process and split in :meth:`setup`
1398+
1399+
However, the above are only necessary for distributed processing.
1400+
1401+
.. warning:: do not assign state in prepare_data
1402+
13871403
13881404
- :meth:`~pytorch_lightning.trainer.Trainer.fit`
13891405
- ...
13901406
- :meth:`prepare_data`
1407+
- :meth:`setup`
13911408
- :meth:`train_dataloader`
13921409
- :meth:`val_dataloader`
13931410
- :meth:`test_dataloader`

pytorch_lightning/trainer/evaluation_loop.py

+20-16
Original file line numberDiff line numberDiff line change
@@ -341,9 +341,10 @@ def _evaluate(
341341
elif self.is_overridden('validation_epoch_end', model=model):
342342
eval_results = model.validation_epoch_end(outputs)
343343

344-
# aggregate ddp stats across
345-
if self.use_ddp or self.use_ddp2:
346-
self.reduce_eval_ddp(eval_results)
344+
# aggregate ddp stats across
345+
has_content = eval_results is not None and len(eval_results) > 0
346+
if has_content and (self.use_ddp or self.use_ddp2):
347+
self.reduce_eval_ddp(eval_results)
347348

348349
# enable train mode again
349350
model.train()
@@ -406,23 +407,26 @@ def run_evaluation(self, test_mode: bool = False):
406407

407408
# run evaluation
408409
eval_results = self._evaluate(self.model, dataloaders, max_batches, test_mode)
409-
_, prog_bar_metrics, log_metrics, callback_metrics, _ = self.process_output(eval_results)
410410

411-
# add metrics to prog bar
412-
self.add_progress_bar_metrics(prog_bar_metrics)
411+
# enable no returns
412+
if eval_results is not None and len(eval_results) > 0:
413+
_, prog_bar_metrics, log_metrics, callback_metrics, _ = self.process_output(eval_results)
413414

414-
# log results of test
415-
if test_mode and self.is_global_zero:
416-
print('-' * 80)
417-
print('TEST RESULTS')
418-
pprint(callback_metrics)
419-
print('-' * 80)
415+
# add metrics to prog bar
416+
self.add_progress_bar_metrics(prog_bar_metrics)
420417

421-
# log metrics
422-
self.log_metrics(log_metrics, {})
418+
# log results of test
419+
if test_mode and self.is_global_zero:
420+
print('-' * 80)
421+
print('TEST RESULTS')
422+
pprint(callback_metrics)
423+
print('-' * 80)
423424

424-
# track metrics for callbacks
425-
self.callback_metrics.update(callback_metrics)
425+
# log metrics
426+
self.log_metrics(log_metrics, {})
427+
428+
# track metrics for callbacks
429+
self.callback_metrics.update(callback_metrics)
426430

427431
# hook
428432
model.on_post_performance_check()

pytorch_lightning/trainer/trainer.py

+5-7
Original file line numberDiff line numberDiff line change
@@ -129,11 +129,6 @@ class Trainer(
129129
>>> trainer.fit(model, train_loader)
130130
1
131131
>>> trainer.test(model, train_loader) # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
132-
--------------------------------------------------------------------------------
133-
TEST RESULTS
134-
...
135-
--------------------------------------------------------------------------------
136-
137132
"""
138133
DEPRECATED_IN_0_9 = ('use_amp', 'show_progress_bar', 'training_tqdm_dict', 'num_tpu_cores')
139134

@@ -1142,8 +1137,11 @@ def run_pretrain_routine(self, model: LightningModule):
11421137
self.val_dataloaders,
11431138
max_batches,
11441139
False)
1145-
_, _, _, callback_metrics, _ = self.process_output(eval_results)
1146-
self.callback_metrics = callback_metrics
1140+
1141+
# allow no returns from eval
1142+
if eval_results is not None and len(eval_results) > 0:
1143+
_, _, _, callback_metrics, _ = self.process_output(eval_results)
1144+
self.callback_metrics = callback_metrics
11471145

11481146
self.on_sanity_check_end()
11491147

0 commit comments

Comments
 (0)