Runtime estimator #1991
Merged
Runtime estimator #1991
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…mvpatel2000/composer into mvpatel2000/bottom-up-runtime-estimator
|
Note: The weird spikes in the wandb graph are because wandb associates the last logged time for a given step, which makes graph weird when x-axis is time. I spent a while investigating -- happy to discuss offline since it's a little too complicated to writeup. As a side effect, @eracah and I discovered where timestamp is advanced for batches vs. epochs compared to when metrics are calculated is inconsistent -- along for the ride I reorder the epoch one to be consistent. Before and after for metrics not changing |
mvpatel2000
dismissed
Glabred’s stale review
Let's discuss offline... we should get you through a starter task and maybe a training project before you're ready
dakinggg
approved these changes
Feb 24, 2023
|
Also, I'm willing to merge this with just manual tests, but please do add some unit tests in a follow on pr |
bcui19
reviewed
Feb 24, 2023
bcui19
approved these changes
Feb 24, 2023
|
Added a test |
hanlint
approved these changes
Feb 24, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Estimates remaining runtime. Uses a simple timer interpolation + correction for eval. Graph shows a "normal" run and a corrected run with eval.
Note that this doesn't always work super smoothly, e.g. resnet estimates are not great because first epoch is a lot slower due to dataloader issues.
We will probably hit more issues as well later on, so this will require corrections.
Import Questions:
wall_clock/is intuitive, but everything else is in seconds so we need to report in seconds (whereas we would prefer to probably report hours?)What issue(s) does this change relate to?
CO-1664