Skip to content

fix(tests): add retries to model checkpoint test#486

Merged
douglas-reid merged 2 commits into
mainfrom
doug/fix-model-checkpoint-save-load-flake
Jul 15, 2023
Merged

fix(tests): add retries to model checkpoint test#486
douglas-reid merged 2 commits into
mainfrom
doug/fix-model-checkpoint-save-load-flake

Conversation

@douglas-reid
Copy link
Copy Markdown
Contributor

@douglas-reid douglas-reid commented Jul 14, 2023

There is a test flake around DEFAULT checkpoints when multiple test runs happen at the same time. This PR is an attempt to solve that by attempting to rerun the checks until the hashes match.

I don't know if that's the best way to handle this issue, but it was all I could think of at the moment. Someone with a deeper understanding of the intended behavior here could possibly come up with a better solution.

Comment thread tests/steamship_tests/plugin/unit/trainable/test_model_checkpoints.py Outdated
@douglas-reid douglas-reid force-pushed the doug/fix-model-checkpoint-save-load-flake branch from c4f33f3 to 9c60c28 Compare July 15, 2023 00:28
@douglas-reid douglas-reid changed the title fix(tests): add hash of files to add robustness to model checkpoint fix(tests): add retries to model checkpoint test Jul 15, 2023
@douglas-reid douglas-reid force-pushed the doug/fix-model-checkpoint-save-load-flake branch from 9c60c28 to a08805d Compare July 15, 2023 00:30
@douglas-reid douglas-reid merged commit 6a34440 into main Jul 15, 2023
@douglas-reid douglas-reid deleted the doug/fix-model-checkpoint-save-load-flake branch July 15, 2023 02:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant