-
Notifications
You must be signed in to change notification settings - Fork 12
Warmstart automation #302
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Warmstart automation #302
Conversation
fromm-m
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
great work!
flxst
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, but I think the end2end test might not be sufficient, see my comment.
flxst
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Added a few comments that should perhaps be addressed, nothing serious though.
| # cd to the directory of the script (absolute pathe) | ||
| cd "$(dirname "$0")" | ||
|
|
||
| rm -rf ../data/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if we should move this line to the end of the shell script, right above the final "Finished warmstart example" message. This would clean up the temporary data right after the test, instead of cleaning up the data of a previous test at the beginning. (This is what is done in the case of the getting started example test as well).
Co-authored-by: Felix Stollenwerk <felix.stollenwerk@ai.se>
Co-authored-by: Felix Stollenwerk <felix.stollenwerk@ai.se>
Co-authored-by: Felix Stollenwerk <felix.stollenwerk@ai.se>
What does this PR do?
This PR simplifies warmstarts from a checkpoint by adding a warmstart entrypoint.
We pass the warmstart config and the checkpoint info file to the endpoint. The checkpoint info file contains the path information for the model and optimizer checkpoint. The checkpoint path is injected into the config via a pydantiv env resolver.
Checklist before submitting final PR
python tests/tests.py)CHANGELOG_DEV.md)