Changed smoothing in tqdm to decrease variability of time remaining

[pertschuk]

between training / eval

Before submitting

• Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you write any new necessary tests?
• Did you make sure to update the docs?
• If you made a notable change (that affects users), did you update the CHANGELOG?

What does this PR do?

Temporary fix for #1096

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

Merging #1194 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #1194   +/-   ##
======================================
  Coverage      91%     91%           
======================================
  Files          62      62           
  Lines        3119    3119           
======================================
  Hits         2828    2828           
  Misses        291     291

[Borda]

@pertschuk could you elaborate why it is temporary? so it does no solve the linked problem?

[pertschuk]

@Borda

So originally, the main_progress_bar had exponential smoothing of 0.3 (tqdm default), causing the time remaining to fluctuate within an epoch depending on whether it was running training or evaluation steps.

This PR changes smoothing to 0.0, so that the estimated time remaining is based an an average of ALL steps thus and fluctuates less represents a more accurate epoch ETA.

But time remaining is still not entirely accurate (especially prior to the first validation run within epoch, as it doesn't know how long these validation steps will be).

I experimented with implementing a custom timer for this (by timing the dummy_eval_steps), but it was a significant addition of code, and didn't work with tqdm.

This is because TQDM assumes that all steps be roughly equivalent time, thus having eval steps and train steps increment the same tqdm iterator (main_progress_bar) is probably not best practice (but it may be okay here for now).

If we were to want it to be 100% accurate we should probably separate out training / eval into separate loops with nested progress bars like such:

In this example there are 5 eval runs, meaning val_check_interval=0.2 and then training would be split into chunks accordingly. So basically the main_progress_bar would have to be separated out into sub-progress bars

Epochs:   0%|          | 0/4 [00:00<?, ?it/s]
Eval Runs:   0%|          | 0/5 [00:00<?, ?it/s]
Training:   0%|          | 0/50 [00:00<?, ?it/s]

And then:

Epochs:   0%|          | 0/4 [00:00<?, ?it/s]
Eval Runs:   0%|          | 0/5 [00:00<?, ?it/s]
Val:   0%|          | 0/50 [00:00<?, ?it/s]

[Borda]

LGTM 🚀