Implementing pytorch-lightning #94
Conversation
…g rate callback along with custom progress bar
…el; print validation loss to commandline when training
… e.g. to have the model predict log10(energy) given targets that are naturally in units of energy
There was a problem hiding this comment.
Nice! Good to see that PyL doesnt change the usage much.
One question; in the original implementation of the PiecewiseLinear Scheduler, it is possible to specify the initial LR (lowest bound), the maximal LR and the ending LR so three points in total. Is this still possible in the PyL implementation?
Rasmus
|
Yes, @RasmusOrsoe! I have implemented a generic piecewise linear learning rate scheduler here, i.e. one where you can specify an arbitrary number of "milestones" (i.e. epoch/step counts) and the corresponding learning rate multipliers for each. Then the scheduler interpolates linearly between these milestones. A special case is then having a starting LR multiplier, an end LR multiplier, and a "peak" somewhere in between. I have included an example of this, that should mirror the original implementation exactly, here. |
|
Tagging @kaareendrup cf. our chat earlier today. |
This PR introduces pytorch-lightning (PyL) to handle the main training loop. This introduces and number of quality-of-life benefits and means that we can reduce maintenance overhead quite substantially since the custom trainer classes and callbacks could be replaced more or less wholesale by PyL's built-in classes. Custom callbacks (e.g. the
PiecewiseLinearLRscheduler) have been reimplemented using PyL's classes and have been confirmed to behave exactly as the original class it is replacing. A new example scripts examples/test_model_training_pytorch_lightning.py shows how to train a model in the PyL framework.However, in the interest of ensuring full reproducibility of the paper results, I have agreed with @RasmusOrsoe to keep all of the old (i.e. non-PyL) model and training code in the src/gnn_reco/legacy/ module for the time being. After the low-energy performance paper is completed, we will tag the repo version and start deprecating these in favour of the functionality in the main branch. The example script examples/test_model_training_sqlite.py shows how to perform energy regression using this legacy code. The two example scripts have been tested and found to yield similar losses during training.
This PR addresses the problem with energy reconstruction pointed out in #88
Closes #76
Tagging @BozianuLeon and @kaareendrup as this might be of interest to you.