[AIR] Remove head node syncing as the default storage option

[justinvyu]

Why are these changes needed?

See the REP and Github Issue for context:

• REP
• Github Issue

Here are the possible scenarios:

1. Single-node experiment: No error raised / no behavior change. Default syncing is disabled by default, which should not change any behavior. Before, we launched a syncing operation, but it was a no-op since it tried to sync a local directory to itself.
2. Multi-node experiment + no checkpointing: A warning will be logged the first time AIR tries to pull the directory of a trial running on a remote node. All metrics get streamed to the driver, so those are still persisted to the head node. This really only affects trial artifacts, which are no longer pulled to the head node.
3. Multi-node experiment + checkpoints + no cloud storage: Error raised upon the first checkpoint of a trial living on a worker node.
4. Multi-node experiment + checkpoints + cloud storage: No error raised / no behavior change.

Error message:

E               DeprecationWarning: Ray AIR no longer supports the synchronization of checkpoints and other artifacts from worker nodes to the head node. This means that the checkpoints and artifacts saved by trials scheduled on worker nodes will not be accessible during the run (e.g., resuming from a checkpoint after a failure) or after the run (e.g., loading the checkpoint of a trial that ran on an already terminated worker node).
E               
E               To fix this issue, configure AIR to use either:
E               (1) Cloud storage: `RunConfig(storage_path='s3://your/bucket')`
E               (2) A network filesystem mounted on all nodes: `RunConfig(storage_path='/mnt/path/to/nfs_storage')`
E               See this Github issue for more details on transitioning to cloud storage/NFS as well as an explanation on why this functionality is being removed: https://github.com/ray-project/ray/issues/37177
E               
E               Other temporary workarounds:
E               - If you want to avoid errors/warnings and continue running with syncing explicitly turned off, set `RunConfig(SyncConfig(syncer=None))`
E               - Or, to re-enable the head node syncing behavior, set the environment variable RAY_AIR_REENABLE_DEPRECATED_SYNC_TO_HEAD_NODE=1
E                 - **Note that this functionality will tentatively be hard-deprecated in Ray 2.7.** See the linked issue for the latest information.

TODO

• Use the env var for any legacy tests
• Fix all broken release tests
• E2E test?
• Link a Github issue explaining this decision

Follow-up PR

• Update docs
  • sync_on_checkpoint is useless if head node syncing is disabled
  • syncer=None is not needed anymore for NFS, since it's disabled by default now

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[matthewdeng]

Do we need the same error message here?

[matthewdeng]

Oh I see, this is for artifact syncing?

[justinvyu]

If a user is running multi-node without checkpointing, but still saves artifacts:

• on_checkpoint does not get called
• Trial artifacts would no longer be pulled to the head node.

We can do one of these options:

1. Detect if the trial wrote any artifacts. If so, raise an error on this callback.
2. Just log a warning on the first time sync_trial_dir gets called for a remote trial and does a no-op, saying something like "Syncing to head node is disabled. If you are writing any trial artifacts, they will not be persisted. Please do X or Y." Logging on every sync_trial_dir will spam the logs too much.

I think option 2 is safer.

[avnishn]

lgtm pending tests passing. Can you run all of the rllib release tests though: name:rllib_.*

it seems you only ran the SAC ones.

[justinvyu]

Ran the full suite of ml and rllib release tests by picking this PR onto the release branch. There were too many confounding factors on master to figure out what failures were caused by my PR vs. not. See here: https://buildkite.com/ray-project/release-tests-pr/builds/45251#_

All errors here are either unrelated to the PR or unstable, and some have been resolved (ex: a problem core PR has been reverted)

cc: @avnishn for the tests, @sofianhnaide for approval