Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure worker synchronization during resume #3369

Merged
merged 6 commits into from
Apr 27, 2023
Merged

Ensure worker synchronization during resume #3369

merged 6 commits into from
Apr 27, 2023

Conversation

abidwael
Copy link
Contributor

This PR ensures

  • We're not accessing keys that don't exist in the progress tracker dictionary.
  • All keys of the progress tracker dictionary are available for initialization.
  • All workers are synced on whether to resume or create a new progress tracker (due to inconsistencies in the output of resume_files_exist)

Copy link
Collaborator

@justinxzhao justinxzhao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

.gitignore Show resolved Hide resolved
ludwig/trainers/trainer.py Show resolved Hide resolved
ludwig/trainers/trainer.py Show resolved Hide resolved
@github-actions
Copy link

Unit Test Results

  6 files  ±  0    6 suites  ±0   45m 53s ⏱️ + 5m 41s
33 tests  - 48  29 ✔️  - 46    4 💤  - 2  0 ±0 
86 runs   - 27  72 ✔️  - 26  14 💤  - 1  0 ±0 

Results for commit d391080. ± Comparison against base commit 2b3db46.

This pull request removes 48 tests.
tests.ludwig.augmentation.test_augmentation_pipeline ‑ test_ray_model_training_with_augmentation_pipeline[preprocessing0-False]
tests.ludwig.augmentation.test_augmentation_pipeline ‑ test_ray_model_training_with_augmentation_pipeline[preprocessing0-True]
tests.ludwig.augmentation.test_augmentation_pipeline ‑ test_ray_model_training_with_augmentation_pipeline[preprocessing0-augmentation_pipeline_ops2]
tests.ludwig.augmentation.test_augmentation_pipeline ‑ test_ray_model_training_with_augmentation_pipeline[preprocessing1-False]
tests.ludwig.augmentation.test_augmentation_pipeline ‑ test_ray_model_training_with_augmentation_pipeline[preprocessing1-True]
tests.ludwig.augmentation.test_augmentation_pipeline ‑ test_ray_model_training_with_augmentation_pipeline[preprocessing1-augmentation_pipeline_ops2]
tests.ludwig.automl.test_base_config ‑ test_dataset_info[dask]
tests.ludwig.automl.test_base_config ‑ test_dataset_info[pandas]
tests.ludwig.automl.test_base_config ‑ test_infer_parquet_types
tests.ludwig.automl.test_base_config ‑ test_is_field_boolean[dask]
…

@abidwael abidwael merged commit 9b02781 into master Apr 27, 2023
10 of 11 checks passed
@abidwael abidwael deleted the keyerror-train branch April 27, 2023 23:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants