Caption on training improvement misleading #840
Labels
Comments
|
It's a totally valid point, thank you for posting it, will solve it ;) |
w4nderlust
added a commit
that referenced
this issue
Aug 22, 2020
Merged
w4nderlust
added a commit
that referenced
this issue
Sep 11, 2020
* Add learning rate graph (#838) * Add learning rate to tensorboard graphs Add learning rate to tensorboard graphs * Fix copy\paste typo * Use the correct learning rate value (current, not original) * Switch from 0 to None * TF2 porting: Add unit test for input feature 'tied' parameter (#837) * feat: add unit test for input feature tied parameters * fix: constructor to support pre-defined encoder * fix: constructor to support pre-defined encoder * fix: constructor to support pre-defined encoder * feat: additional features for unit tests * doc: cleaned up comments * refactor: account for subclassing for encoder initialization * refactor: add explicit parameter for tied, facilitates positive and negative testing * refactor: micro level unit test function name * feat: add macro level test for tied parameter * Class parameters cleanup * Fixed audio and timeseries features * Fixed timeseries get_input_shape return of max_sequence_length Co-authored-by: w4nderlust <w4nderlust@gmail.com> * Fix #840: Better messaging for plateau logic. * Fix #840: Improved messaging for plateau logic. * Fixing collect weights (#836) Co-authored-by: Travis Addair <taddair@uber.com> * TF2 porting: Conversion of feed forward attention reducer to TF2 (#843) * refactor: initial setup for TF2 * refactor: add training, mask parameters, correct dimension for 2nd dense layer * feat: add unit test for attention modules * style: minor code reformatting * fix: temporary work-around * refactor: incorporate reviewers comments * refactor: revised scope of parameters to test * refactor: incorporate TF2 version of feed forward attention reducer * feat: reduction unit tests * feat: incorporate reduce output setup in embed encoder * feat: named layers in attention reducer * refactor: make generic reduce mode to support both input and output reduction feat: add sequence list reduction * refactor: make use of generic reduction mode parameter * refactor: convert to use SequenceReducerMixin * fix: add missing key to fix error in text_feature() * doc: comment updates * refactor: converted from Mixin to SequenceReducer/SequenceListReducer * refactor: converted from Mixin to SequenceReducer/SequenceListReducer * fix: correct parameter usage for sequence reduction * refactor: convert to SequenceReducer class * refactor: convert to SequenceReducer class * refactor: convert to SequenceReducer class * refactor: remove obsolete code * doc: remove obsolete comments * refactor: remove deprecated function * fix: error in constructor and handling multiple sequence list during call() * refactor: remove SequenceListReducer class and unit test * refactor: removed dependencies reduction test * refactor: add reduce_dependencies parameters to test suite * refactor: updated to test different dependency reductions * fix: issue in calculating the expected hidden size * feat: add attention to list of reduce_dependencies * refactor: remove extraneous reduction key * fix: with 'attention' ensue both dependencies have same hidden size * fix: error in serval unit test due to use of obsolete reduction specifier * doc: updated skip test message * refactor: add explicit clean-up * Minor refactoring class FeedForwardAttentionReducer.call() * Transformed all reducers in classes * Fixed reducer objects * Fixed test dependencies Co-authored-by: w4nderlust <w4nderlust@gmail.com> * Removed todos * Solved optimizers todos * Cleanup losses * Fixed savedmodel test * feat: enable eager execution mode with --debug flag (#849) * Fixing the regularization losses test (#851) * Fixed saved model API to no longer require providing fake outputs as input (#853) * Fix #850: Switch from ValueError to Warning to make stratify work on non-output features (#854) * Switch from ValueError to Warning to make stratify work on non-output features * Revert one to be valueerror and better explain warning message * Improved stratify checks Co-authored-by: w4nderlust <w4nderlust@gmail.com> * Fix ##844: Load LudwigModel in test_savedmodel before creating saved model (#855) * Load LudwigModel in test_savedmodel before creating saved model * Test with and without loading the model * Update TF2 and TFA with latest version (#842) * Adding print for the import error in serve.py * Pin TensorFlow < 2.3 until saved model tests are resolved (#859) * Integrate Hugging Face Transformers (#698) * Added text encoders containing Hugging Face transformers integration * Added HFTokenizer and refactored sequence tokenization (text, category, set and bag to come next) * Adding transformers dependency * Remove tf1 comapt from sequence feature * removed __init__.py file in tests/ludwig to fix test_horovod_utils.py * HF Transformers Integration (#831) * initial changes for hf integration * first pass at complete HF encoder integration * added trainable to text encoder init * added hf encoders integration to text feature * minor edit to BERTEncoder * Further edits to HF transformers integration (#845) * initial changes for hf integration * first pass at complete HF encoder integration * added trainable to text encoder init * added hf encoders integration to text feature * minor edit to BERTEncoder * remove hf transformer logic from seq. features * minor edits to string_utils * minor edits to sequence feature and string utils * fixed bug in TextInputFeature init and removed Bart Model and DialoGPT * minor edit to text_feature * Cleanup strings utils * Fixed savedmodel test and text encoders with reducer classes * HF encoder pre-processing (#856) * First attempt at fixed_encoder_preprocessing * added if/else wrapper to protect for case where key doesn't exist in dict * added fixed_preprocessing_parameters attribute to all encoder classes * added logic to build_data and build_metadata for supporting fixed pre-processing params * Added populating default params for encoders * added capability of resizing HF token embeddings: needed when PAD token is added to pre-trained vocab * parameterized new integration tests using pytest.mark * modified encoder input to support TF 2.0 formatting * commenting out tranformer_xl and flaubert * removed transformer_xl and flaubert from HF_encoders list in utils.py * travis testing * more wandb test debugging * wandb testing - removing transformer imports * moved transformer imports inside relevant functions and added longformer integration * fixed bug Co-authored-by: w4nderlust <w4nderlust@gmail.com> * HF Encoder Debugging (#860) * First attempt at fixed_encoder_preprocessing * added if/else wrapper to protect for case where key doesn't exist in dict * added fixed_preprocessing_parameters attribute to all encoder classes * added logic to build_data and build_metadata for supporting fixed pre-processing params * Added populating default params for encoders * added capability of resizing HF token embeddings: needed when PAD token is added to pre-trained vocab * parameterized new integration tests using pytest.mark * modified encoder input to support TF 2.0 formatting * commenting out tranformer_xl and flaubert * removed transformer_xl and flaubert from HF_encoders list in utils.py * travis testing * more wandb test debugging * wandb testing - removing transformer imports * moved transformer imports inside relevant functions and added longformer integration * fixed bug * fixed T5 encoder and typo in HF_ENCODERS list in utils.py Co-authored-by: w4nderlust <w4nderlust@gmail.com> * Separate HF tests in 2 separate ones, commenting out the full test for speed reasons Co-authored-by: ANarayan <gostanfordtennis@gmail.com> * Increased version number to 0.3.0-a.4 * Added @slow decorator for slow tests * Re-added flaubert to text encoders registry * Upgraded to horovod==0.20.0 (#862) * Fixed TF 2.3 compatibility by not passing None values as layer input during symbolic model construction (#864) * Added multi-version testing to Travis CI (#866) * Added multi-version testing to Travis CI * Fixed typo * Fixed modification in place * Remove tensorflow extra in CI * Remove upper bound on TensorFlow * Fixed beam search decoding to use predictions from FinalBeamSearchDecoderOutput (#867) * TF2 porting: Resolving TF2 todos for sequence decoder (#852) * refactor: modify how sequence lengths are calculated and used in the Tagger decoder * refactor: moved sequence length computation from encoder to sequence input feature * refactor: move location for casting target dtype * refactor: change such that values for END_SYMBOL <> PAD * refactor: added length sequence computation * refactor: go back to END_SYMBOL = PAD = 0 implementation * fix: missing constant for PAD_TOKEN * fix: inconsistent sequence size return during training * fix: mask length computation to include only one <PAD> token * fix: tf.concat() error by expanding predictions/logits to max seq length * refactor: revert fix for padding predictions/logits to tf.concat() issue * refactor: pad predictions/logits to ensure all batch sequences same size * fix: error in graph execution on length manipulation * fix: ensure predictions/logits sequence size are consistent fix: missing logit masking fix: missing setup for embedding matrix * fix: calculation error for offset to last predicted * refactor: add missing LENGTHS element to predictions dictionary * refactor: use lengths to generate predicted sequence output * fix: error after merge with master * fix: error in tagger unit test * fix: sequence length to support both tagger and generator * Fixing lengths, loss and greedy sequence generation. * refactor: reconcile upstream/master merge discrepancy * fix: test_regularizers to account for new handling of lengths * Removed superfluous cast * Fix: logits.shape to tf.shape(logits) Co-authored-by: w4nderlust <w4nderlust@gmail.com> * Fix #833: loads the model after training and before predicting if the model was saved on disk. (#861) * Fix beam search (#868) * Implementing function to obtain logprobs from raw logprobs * Fix sequence loss and perplexity calculation for sequence generation (uses probs instead of logits) * Added lots of comments and cleaned up the probabilities reconstruction in the beam search sampling function * Replaced concat ith stack when reshaping logporbs in beam search decoder * Moved nasty code in beam_search_decode into a new extract_sequence_probabilities function Co-authored-by: Carlo Grisetti <carlo.grisetti@dsgroup.it> Co-authored-by: Jim Thompson <jimthompson5802@gmail.com> Co-authored-by: w4nderlust <w4nderlust@gmail.com> Co-authored-by: Piero Molino <piero@uber.com> Co-authored-by: ANarayan <gostanfordtennis@gmail.com>
w4nderlust
added a commit
that referenced
this issue
Sep 18, 2020
* Moved initialize_batcher in the batcher module * moved batch_predict batch_evaluate and other model specific methods to ECD * Added default None to horovod parameter in batch_predict * Added new_ludwig_model and modified trainer to receive the parameters at init time and recevie just data and model at train time. * Added init save and load code to NewLudwigModel * Updated train_pseudo and implemented an almost working train in NewLudwigModel * Moved prediction function to prediction_helpers and renamed test -> evaluate * Moved prediction function to prediction_helpers and renamed test -> evaluate * Added predict and evaluate to NewLudwigModel * Fix validation_metric to self._validation_metric reference. * Fixed calculate_overall_stats in postprocessing * Renamed feature_metadata to metadata * Adapted calculate_overall_stats for text feature * refactored postprocess_results * Fixed NewLudwigModel.evaluate() postprocessing * renamed postprocess_results -> postprocess_predictions and made it non static * Made calculate_overall_stats non static * Fixes postprocess in NewLudwigModel.predict() * removed evaluate_performance parameter from predict() * Heavy refactoring of preprocessing.py * Applied registry pattern to postprocessing * Fixed evaluation preprocessing in NewLudwigModel * Fixed evaluation preprocessing in NewLudwigModel * merge master * Cleanup + renaming * Added function to figure out data_format automatically * Introduced new Predictor class for managing batch prediction and evaluation * clean up Trainer and moved batch_collect_activations to Predictor * Merge master * Introduce train_cli * Adding data_format paramter to train parseargs * Introduced predict_cli * Introduced evaluate_cli * Adapted collect.py * Removed circular dependence in feature_utils * Merged upstream (#869) * Add learning rate graph (#838) * Add learning rate to tensorboard graphs Add learning rate to tensorboard graphs * Fix copy\paste typo * Use the correct learning rate value (current, not original) * Switch from 0 to None * TF2 porting: Add unit test for input feature 'tied' parameter (#837) * feat: add unit test for input feature tied parameters * fix: constructor to support pre-defined encoder * fix: constructor to support pre-defined encoder * fix: constructor to support pre-defined encoder * feat: additional features for unit tests * doc: cleaned up comments * refactor: account for subclassing for encoder initialization * refactor: add explicit parameter for tied, facilitates positive and negative testing * refactor: micro level unit test function name * feat: add macro level test for tied parameter * Class parameters cleanup * Fixed audio and timeseries features * Fixed timeseries get_input_shape return of max_sequence_length Co-authored-by: w4nderlust <w4nderlust@gmail.com> * Fix #840: Better messaging for plateau logic. * Fix #840: Improved messaging for plateau logic. * Fixing collect weights (#836) Co-authored-by: Travis Addair <taddair@uber.com> * TF2 porting: Conversion of feed forward attention reducer to TF2 (#843) * refactor: initial setup for TF2 * refactor: add training, mask parameters, correct dimension for 2nd dense layer * feat: add unit test for attention modules * style: minor code reformatting * fix: temporary work-around * refactor: incorporate reviewers comments * refactor: revised scope of parameters to test * refactor: incorporate TF2 version of feed forward attention reducer * feat: reduction unit tests * feat: incorporate reduce output setup in embed encoder * feat: named layers in attention reducer * refactor: make generic reduce mode to support both input and output reduction feat: add sequence list reduction * refactor: make use of generic reduction mode parameter * refactor: convert to use SequenceReducerMixin * fix: add missing key to fix error in text_feature() * doc: comment updates * refactor: converted from Mixin to SequenceReducer/SequenceListReducer * refactor: converted from Mixin to SequenceReducer/SequenceListReducer * fix: correct parameter usage for sequence reduction * refactor: convert to SequenceReducer class * refactor: convert to SequenceReducer class * refactor: convert to SequenceReducer class * refactor: remove obsolete code * doc: remove obsolete comments * refactor: remove deprecated function * fix: error in constructor and handling multiple sequence list during call() * refactor: remove SequenceListReducer class and unit test * refactor: removed dependencies reduction test * refactor: add reduce_dependencies parameters to test suite * refactor: updated to test different dependency reductions * fix: issue in calculating the expected hidden size * feat: add attention to list of reduce_dependencies * refactor: remove extraneous reduction key * fix: with 'attention' ensue both dependencies have same hidden size * fix: error in serval unit test due to use of obsolete reduction specifier * doc: updated skip test message * refactor: add explicit clean-up * Minor refactoring class FeedForwardAttentionReducer.call() * Transformed all reducers in classes * Fixed reducer objects * Fixed test dependencies Co-authored-by: w4nderlust <w4nderlust@gmail.com> * Removed todos * Solved optimizers todos * Cleanup losses * Fixed savedmodel test * feat: enable eager execution mode with --debug flag (#849) * Fixing the regularization losses test (#851) * Fixed saved model API to no longer require providing fake outputs as input (#853) * Fix #850: Switch from ValueError to Warning to make stratify work on non-output features (#854) * Switch from ValueError to Warning to make stratify work on non-output features * Revert one to be valueerror and better explain warning message * Improved stratify checks Co-authored-by: w4nderlust <w4nderlust@gmail.com> * Fix ##844: Load LudwigModel in test_savedmodel before creating saved model (#855) * Load LudwigModel in test_savedmodel before creating saved model * Test with and without loading the model * Update TF2 and TFA with latest version (#842) * Adding print for the import error in serve.py * Pin TensorFlow < 2.3 until saved model tests are resolved (#859) * Integrate Hugging Face Transformers (#698) * Added text encoders containing Hugging Face transformers integration * Added HFTokenizer and refactored sequence tokenization (text, category, set and bag to come next) * Adding transformers dependency * Remove tf1 comapt from sequence feature * removed __init__.py file in tests/ludwig to fix test_horovod_utils.py * HF Transformers Integration (#831) * initial changes for hf integration * first pass at complete HF encoder integration * added trainable to text encoder init * added hf encoders integration to text feature * minor edit to BERTEncoder * Further edits to HF transformers integration (#845) * initial changes for hf integration * first pass at complete HF encoder integration * added trainable to text encoder init * added hf encoders integration to text feature * minor edit to BERTEncoder * remove hf transformer logic from seq. features * minor edits to string_utils * minor edits to sequence feature and string utils * fixed bug in TextInputFeature init and removed Bart Model and DialoGPT * minor edit to text_feature * Cleanup strings utils * Fixed savedmodel test and text encoders with reducer classes * HF encoder pre-processing (#856) * First attempt at fixed_encoder_preprocessing * added if/else wrapper to protect for case where key doesn't exist in dict * added fixed_preprocessing_parameters attribute to all encoder classes * added logic to build_data and build_metadata for supporting fixed pre-processing params * Added populating default params for encoders * added capability of resizing HF token embeddings: needed when PAD token is added to pre-trained vocab * parameterized new integration tests using pytest.mark * modified encoder input to support TF 2.0 formatting * commenting out tranformer_xl and flaubert * removed transformer_xl and flaubert from HF_encoders list in utils.py * travis testing * more wandb test debugging * wandb testing - removing transformer imports * moved transformer imports inside relevant functions and added longformer integration * fixed bug Co-authored-by: w4nderlust <w4nderlust@gmail.com> * HF Encoder Debugging (#860) * First attempt at fixed_encoder_preprocessing * added if/else wrapper to protect for case where key doesn't exist in dict * added fixed_preprocessing_parameters attribute to all encoder classes * added logic to build_data and build_metadata for supporting fixed pre-processing params * Added populating default params for encoders * added capability of resizing HF token embeddings: needed when PAD token is added to pre-trained vocab * parameterized new integration tests using pytest.mark * modified encoder input to support TF 2.0 formatting * commenting out tranformer_xl and flaubert * removed transformer_xl and flaubert from HF_encoders list in utils.py * travis testing * more wandb test debugging * wandb testing - removing transformer imports * moved transformer imports inside relevant functions and added longformer integration * fixed bug * fixed T5 encoder and typo in HF_ENCODERS list in utils.py Co-authored-by: w4nderlust <w4nderlust@gmail.com> * Separate HF tests in 2 separate ones, commenting out the full test for speed reasons Co-authored-by: ANarayan <gostanfordtennis@gmail.com> * Increased version number to 0.3.0-a.4 * Added @slow decorator for slow tests * Re-added flaubert to text encoders registry * Upgraded to horovod==0.20.0 (#862) * Fixed TF 2.3 compatibility by not passing None values as layer input during symbolic model construction (#864) * Added multi-version testing to Travis CI (#866) * Added multi-version testing to Travis CI * Fixed typo * Fixed modification in place * Remove tensorflow extra in CI * Remove upper bound on TensorFlow * Fixed beam search decoding to use predictions from FinalBeamSearchDecoderOutput (#867) * TF2 porting: Resolving TF2 todos for sequence decoder (#852) * refactor: modify how sequence lengths are calculated and used in the Tagger decoder * refactor: moved sequence length computation from encoder to sequence input feature * refactor: move location for casting target dtype * refactor: change such that values for END_SYMBOL <> PAD * refactor: added length sequence computation * refactor: go back to END_SYMBOL = PAD = 0 implementation * fix: missing constant for PAD_TOKEN * fix: inconsistent sequence size return during training * fix: mask length computation to include only one <PAD> token * fix: tf.concat() error by expanding predictions/logits to max seq length * refactor: revert fix for padding predictions/logits to tf.concat() issue * refactor: pad predictions/logits to ensure all batch sequences same size * fix: error in graph execution on length manipulation * fix: ensure predictions/logits sequence size are consistent fix: missing logit masking fix: missing setup for embedding matrix * fix: calculation error for offset to last predicted * refactor: add missing LENGTHS element to predictions dictionary * refactor: use lengths to generate predicted sequence output * fix: error after merge with master * fix: error in tagger unit test * fix: sequence length to support both tagger and generator * Fixing lengths, loss and greedy sequence generation. * refactor: reconcile upstream/master merge discrepancy * fix: test_regularizers to account for new handling of lengths * Removed superfluous cast * Fix: logits.shape to tf.shape(logits) Co-authored-by: w4nderlust <w4nderlust@gmail.com> * Fix #833: loads the model after training and before predicting if the model was saved on disk. (#861) * Fix beam search (#868) * Implementing function to obtain logprobs from raw logprobs * Fix sequence loss and perplexity calculation for sequence generation (uses probs instead of logits) * Added lots of comments and cleaned up the probabilities reconstruction in the beam search sampling function * Replaced concat ith stack when reshaping logporbs in beam search decoder * Moved nasty code in beam_search_decode into a new extract_sequence_probabilities function Co-authored-by: Carlo Grisetti <carlo.grisetti@dsgroup.it> Co-authored-by: Jim Thompson <jimthompson5802@gmail.com> Co-authored-by: w4nderlust <w4nderlust@gmail.com> Co-authored-by: Piero Molino <piero@uber.com> Co-authored-by: ANarayan <gostanfordtennis@gmail.com> * Fixed various issues related to hyperopt testing (#871) * Fixed various issues related to hyperopt testing * Added back logger global * Fixed API classification tests (#872) * Fixed save hyperparameters and weights in NewLudwigModel * refactor: make method name consistent with current design * refactor: initial refactor to support new LudwigModel * refactor: adjust for method signatures from refactoring * refactor: more changes to account for recent refactoring work * Fixed merge issue by restoring usage of postprocess_predictions (#873) * Fixed merge issue by restoring usage of postprocess_predictions * Commented out model.save_definition calls until we have a solution in place * TF2: Fixing up experiment to support trainer.py refactoring (#874) * refactor: incorporate reviewer comment and clean out redundant code * refactor: function name renaming * Fixed horovod tests (#877) * Added todos * Fixed collect_weights and collect_activations (#875) * Fixed collect_weights * Fixed collect_activations * Fixed comet and wandb tests (#876) * Fixed comet tests * Fixed wandb tests * Renamed NewLudwigModel to LudwigModel and moved it to api,py * Added and cleaned refactoring todos * Removed api.kfold_cross_validate as the function exists in experiment.py (#878) Co-authored-by: Piero Molino <piero@uber.com> * fix: import error that prevents unit tests from running (#880) * fix: import error that prevented unit tests from running * fix: use updated parameter name * Fixed inference related tests (#881) * Fixed model save and load * Fixed neuropod * Fixed test_reducers * Fixed saved model * Fixed test_server * TF2: kfold_cv refactoring in support of model trainer refactoring (#879) * refactor: incorporate reviewer comment and clean out redundant code * refactor: function name renaming * refactor: change how predictions are saved and return signature * refactor: ajdust for refatoring of LudwigModel * refactor: simplify output feature calculate_overall_stats() signature * test: added test case for text output feature * Moved kfold_cross_validate to api.py * incorporated reviewers comments * refactor: adapted kfold_cross_validate to api.py Co-authored-by: Piero Molino <w4nderlust@gmail.com> * Refactored calculate_overall_statistics to pull in optional instance metadata (#882) * Fixed several visualization tests (#883) * Fixed usage of skip_save_predictions and skip_save_eval_stats * Fixed saving predictions for Pandas data format * TF2 fix experiment unit test (#884) * refactor: incorporate reviewer comment and clean out redundant code * refactor: function name renaming * refactor: change how predictions are saved and return signature * refactor: ajdust for refatoring of LudwigModel * refactor: simplify output feature calculate_overall_stats() signature * test: added test case for text output feature * fix: update parameter usage to refactored trainer model * refactor: remove duplicate code * Fixed get_experiment_description * Divided postprocessing into two steps so predictions will always be saved while in dict form * Plumb through skip_save_unprocessed_output * Removed debug print * Fixed collecting predictions * Fixed more predictions usage * Fixed all visualization API tests * Fixed test_regularizers * Forward args from LudwigModel to Trainer * Fixed typo * refactor: parametrize image unit test * Updated docstrings * Fixed image inputs in_memory issue * Cleanup * Fixed test_experiment_model_resume * Fixed test_tied_macro_level * Added RuntimeError exception to horovod _nccl_available function. * test: added image dataset unit tests * Modifying the logic for exp_dir_name * Fix hyperopt * Add output_dir from predict when removing dirs in test_api.py * Fixed evaluate problem with output dir creation and neuropod error * test: refactor image data set test support different combinations * fix test-server and test_zisualization_api * Fixed experiment * Fix kfold cv * Fix test_model_save_and_load * Fix test_model_training_options * test: corrected for df-hdf5 combination * Fixed serve * Fixed neuropod * Fixed savedmodel * test: add in_memory options and resequence code to prevent overwrite issue * test: override in_memory when source dataset is a dataframe * Removed TODOs * Removed test code in api * Fixed loading best weights after training and added support for distributed training * Fixed Horovod tests * Fixed loading model weights * Adopted output_dir * Removed get_connected_model during load * Fixed configure_horovod * test: refactor code on where to override in_memory flag * refactor: adjust code to override in_memory flag * test: special handling when creating hdf5 data for prediction testing * test: code clean-up * Added back LudwigModel.train_online * Fixed output_dir * Removed output_dir * Fixed collect_activations to not include the outputs * Fix errors when hdf5 is used for prediction * Fixed savedmodel tests error by moving the statement that gets DATA_TRAIN_HDF5_FP after loading metadata Co-authored-by: Travis Addair <taddair@uber.com> Co-authored-by: Carlo Grisetti <carlo.grisetti@dsgroup.it> Co-authored-by: Jim Thompson <jimthompson5802@gmail.com> Co-authored-by: ANarayan <gostanfordtennis@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello :)
This piece of code
https://github.com/uber/ludwig/blob/62430e4a0dd7a4fda08d6dcd615fbdbbf53c5377/ludwig/models/trainer.py#L1316-L1318
although procedurally correct (I want to track whether there has been an improvement since the last plateau and learning rate reduction, to look for the next plateau) leads to a technically wrong statement in the CLI output.
Let's assume that the model is training, 5 epochs have passed and the learning rate gets automatically dropped as per the
reduce_learning_rate_on_plateauparameters. The "last improvement epoch" gets reset to the current epoch (to keep track of the fact that we reduced the learning rate, and we thus need to wait another 5 epochs without improvement before reducing the learning rate again), but in the CLI the next epoch will show that it's only 1 epoch since the last improvement (which should have been 6), the next one will show 2 epochs (but it should be 7), and this only gets fixed as soon as we have an actual improvement.It's trivial, but maybe it gets to your nerves as much as it does to me :)
The text was updated successfully, but these errors were encountered: