-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[train] New persistence mode: Add storage type telemetry #39286
Conversation
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
Signed-off-by: Justin Yu <justinvyu@anyscale.com>
…age_telemetry Signed-off-by: Justin Yu <justinvyu@anyscale.com>
- 'custom' = All other storage schemes, which includes ALL cases where a | ||
custom `storage_filesystem` is provided. | ||
""" | ||
whitelist = {"local", "mock", "s3", "gcs", "abfs", "hdfs"} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there also gs
as an alternative for gcs
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nope, pyarrow fs type name is always gcs
if _use_storage_context(): | ||
air_usage.tag_storage_type(experiments[0].storage) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How come this is tracked down here rather than where the TODO was?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I need the Experiment
object to be initialized to access the StorageContext
…#39286) Signed-off-by: Justin Yu <justinvyu@anyscale.com>
…#39286) Signed-off-by: Justin Yu <justinvyu@anyscale.com>
…39368) * fix tune_hvd_keras (#39223) Signed-off-by: Justin Yu <justinvyu@anyscale.com> * [train] New persistence mode: Add backwards compatibility support for `local_dir` (#39282) Signed-off-by: Justin Yu <justinvyu@anyscale.com> * [train] Add TrainContext.get_storage (#39281) Signed-off-by: Justin Yu <justinvyu@anyscale.com> * [train] New persistence mode: Deprecate experimental distributed checkpointing configs (#39279) Signed-off-by: Justin Yu <justinvyu@anyscale.com> * [Release] Fix `air_example_dolly_v2_lightning_fsdp_finetuning` with large cpu mem head node (#39263) Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com> * Remove all "ray.init" cell output in the example notebooks. (#39283) Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com> * [train] New persistence mode: Add storage type telemetry (#39286) Signed-off-by: Justin Yu <justinvyu@anyscale.com> * [air-doc] Rework experiment tracking docs for Torch trainer. (#38684) * Rework experiment tracking for DDP trainer. Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com> * [docs] add Developer Guides landing page (#39296) Signed-off-by: Matthew Deng <matt@anyscale.com> * [2.7][Doc] Clean up more Ray Train examples (#39284) Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com> * disable Train examples with authentication for now. (#39358) Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com> * [air] move doc_code and examples into respective libraries (#39298) Signed-off-by: Matthew Deng <matt@anyscale.com> --------- Signed-off-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com> Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com> Signed-off-by: Matthew Deng <matt@anyscale.com> Co-authored-by: Justin Yu <justinvyu@anyscale.com> Co-authored-by: Yunxuan Xiao <yunxuanx@anyscale.com> Co-authored-by: xwjiang2010 <87673679+xwjiang2010@users.noreply.github.com>
…#39286) Signed-off-by: Justin Yu <justinvyu@anyscale.com>
…#39286) Signed-off-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: Jim Thompson <jimthompson5802@gmail.com>
…#39286) Signed-off-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: Victor <vctr.y.m@example.com>
Why are these changes needed?
This adds back storage telemetry for the new codepath. Only a whitelist of default pyarrow implementations will be tracked. All custom filesystems will be marked as "custom".
Related issue number
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/
under thecorresponding
.rst
file.