[tune] Automatically detect/activate proper SyncConfig given autoscaler

[richardliaw]

Describe your feature request

This is a common failure mode.

def train_func(config, checkpoint_dir):
    ...
    with tune.checkpoint_dir(...) as ckpt:
       ...
    tune.report()

One common problem is that the user is using K8s or Docker with the autoscaler, and they do not remember to set DockerSyncer or KubernetesSyncer in the SyncConfig.

Instead, we should automatically detect the autoscaler configuration presence and activate these syncers without the user knowing about it.

@krfricke

cc @mkoh-asapp @richardrl

[richardliaw]

This requires a utility function for identifying whether it's using an autoscaling cluster.

[mkoh-asapp]

I missed the part in the docs explaining this so I had to have Richard explain it to me. But even after I set the syncer on tune.run, it wasn't working for me. I realized that this is because we are passing Experiment directly to tune.run, instead of passing a function or class. Once I set the syncer on the Experiments, then it worked, but that was not very clear from the docs.

Maybe it isn't a standard way to use tune (creating Experiments manually), but it might be nice to have that explained somewhere.

Just wanted to bring up a point to consider. Thanks 🎉

[bllchmbrs]

does setting sync_to_driver=False in tune.run(...) silence the error?

The answer to 👆 is no

[krfricke]

Closed via #12108