[air] New train.Checkpoint API: Update release tests (batch 2)
#38550
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com> Fix lint 2 Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com> missing Signed-off-by: Justin Yu <justinvyu@anyscale.com>
…vanilla tf exactly Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
justinvyu
requested review from
a team,
richardliaw,
gjoliver,
krfricke,
xwjiang2010,
amogkam,
matthewdeng,
Yard1 and
maxpumperla
as code owners
aslonnie
reviewed
Aug 17, 2023
|
@aslonnie The CI changes here are temporary just to launch a subset of release tests. Going to revert these before merging, so no need to review here. |
This reverts commit f1b9534.
…update_release_tests_batch2
This reverts commit e6608e8.
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
justinvyu
commented
Aug 17, 2023
Comment on lines
+556
to
+562
|
|
||
| # Raise an error if the storage path is not accessible when | ||
| # attempting to upload a checkpoint from a remote worker. | ||
| # Ex: If storage_path is a local path, then a validation marker | ||
| # will only exist on the head node but not the worker nodes. | ||
| self._check_validation_file() | ||
|
|
There was a problem hiding this comment.
Note: This is a change that's needed so that multi-node w/ no storage path doesn't fail when NO checkpoints are reported.
There was a problem hiding this comment.
@ericl This is the same raise error behavior that we added for head node syncing in 2.6.
arvind-chandra
pushed a commit
to lmco/ray
that referenced
this pull request
Aug 31, 2023
…-project#38550) This PR enables the new persistence mode feature flag for the following release tests: workspace_template_many_model_training air_benchmark_data_bulk_ingest air_benchmark_tensorflow_mnist_cpu_1x4 air_benchmark_tensorflow_mnist_cpu_4x1 air_benchmark_tensorflow_mnist_cpu_4x4 air_benchmark_tensorflow_mnist_gpu_4x4 air_example_dreambooth_finetuning alpa_opt_2_7b_sanity_check alpa_opt_30b_inference cluster_tune_scale_up_down long_running_many_ppo ml_user_horovod_user_test_latest ml_user_horovod_user_test_master ml_user_train_tensorflow_mnist_test ml_user_train_torch_linear_test ml_user_tune_rllib_connect_test train_horovod_multi_node_test Signed-off-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
vymao
pushed a commit
to vymao/ray
that referenced
this pull request
Oct 11, 2023
…-project#38550) This PR enables the new persistence mode feature flag for the following release tests: workspace_template_many_model_training air_benchmark_data_bulk_ingest air_benchmark_tensorflow_mnist_cpu_1x4 air_benchmark_tensorflow_mnist_cpu_4x1 air_benchmark_tensorflow_mnist_cpu_4x4 air_benchmark_tensorflow_mnist_gpu_4x4 air_example_dreambooth_finetuning alpa_opt_2_7b_sanity_check alpa_opt_30b_inference cluster_tune_scale_up_down long_running_many_ppo ml_user_horovod_user_test_latest ml_user_horovod_user_test_master ml_user_train_tensorflow_mnist_test ml_user_train_torch_linear_test ml_user_tune_rllib_connect_test train_horovod_multi_node_test Signed-off-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: Victor <vctr.y.m@example.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
This PR enables the new persistence mode feature flag for the following release tests:
workspace_template_many_model_training
air_benchmark_data_bulk_ingest
air_benchmark_tensorflow_mnist_cpu_1x4
air_benchmark_tensorflow_mnist_cpu_4x1
air_benchmark_tensorflow_mnist_cpu_4x4
air_benchmark_tensorflow_mnist_gpu_4x4
air_example_dreambooth_finetuning
alpa_opt_2_7b_sanity_check
alpa_opt_30b_inference
cluster_tune_scale_up_down
long_running_many_ppo
ml_user_horovod_user_test_latest
ml_user_horovod_user_test_master
ml_user_train_tensorflow_mnist_test
ml_user_train_torch_linear_test
ml_user_tune_rllib_connect_test
train_horovod_multi_node_test
See them passing here (I removed the dolly one from the PR for now): https://buildkite.com/ray-project/release-tests-pr/builds/49414#_
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.