Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Load best weights outside of finally block, since load may throw an exception #2268

Merged

Conversation

dantreiman
Copy link
Contributor

@dantreiman dantreiman commented Jul 12, 2022

The problem here is that if model training fails before the weights are saved, the finally block will call self.model.load(save_path) which throws a FileNotFound exception, thus the original error is never printed to the console.

Python executes the finally block before raising the original exception.

This PR moves self.model.load(save_path) outside of the finally block.

@github-actions
Copy link

Unit Test Results

       4 files   -     1         4 suites   - 1   1h 20m 49s ⏱️ - 39m 37s
2 929 tests ±    0  2 883 ✔️ ±    0  46 💤 ±  0  0 ±0 
8 445 runs   - 171  8 349 ✔️  - 148  96 💤  - 23  0 ±0 

Results for commit dcd62e6. ± Comparison against base commit 9c58d5e.

@dantreiman dantreiman merged commit 740148a into ludwig-ai:master Jul 13, 2022
@dantreiman dantreiman deleted the daniel/report_training_exception branch July 13, 2022 17:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants