[tune] Improve excessive syncing warning and make some deprecations

[justinvyu]

Why are these changes needed?

This PR makes it so that the "Experiment state snapshotting has been triggered multiple..." warning message is less confusing and does not always trigger at the end of training.

• Previously, the wording made it seem like this was an error, which makes it confusing when it's printed out along with an actual checkpointing error.
• The warning would also often happen at the end of training, since we always trigger a forced driver sync at the end of a run, where a previous sync may have happened really recently. For example, a simple script like this will print out the error:

from ray import tune

kwargs = {"kwarg1": 1, "kwarg2": 2}


def train_fn(config, **trainable_kwargs):
    print(config, trainable_kwargs)


tune.Tuner(tune.with_parameters(train_fn, **kwargs)).fit()

Trial train_fn_a9462_00000 completed after 0 iterations at 2024-05-08 15:38:18. Total running time: 1s
2024-05-08 15:38:18,757 WARNING experiment_state.py:205 -- Experiment state snapshotting has been triggered multiple times in the last 5.0 seconds. A snapshot is forced if `CheckpointConfig(num_to_keep)` is set, and a trial has checkpointed >= `num_to_keep` times since the last snapshot.
You may want to consider increasing the `CheckpointConfig(num_to_keep)` or decreasing the frequency of saving checkpoints.
You can suppress this error by setting the environment variable TUNE_WARN_EXCESSIVE_EXPERIMENT_CHECKPOINT_SYNC_THRESHOLD_S to a smaller value than the current threshold (5.0).
2024-05-08 15:38:18,758 INFO tune.py:1007 -- Wrote the latest version of all result files and experiment state to '/Users/justin/ray_results/train_fn_2024-05-08_15-38-13' in 0.0031s.

This PR also makes some deprecations: TUNE_RESULT_DIR, RAY_AIR_LOCAL_CACHE_DIR, local_dir

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[matthewdeng]

[matthewdeng]

My first thought is that it might be odd to raise if an environment variable is set, but I'm not sure what the best practice is, so this is probably fine?

[justinvyu]

I was considering keeping it a warning for another release before just removing it.

Pros:

• No breaking user code

Cons:

• Implicit behavior change that users may be relying on will happen without warning / with no suggested fix

[woshiyyya]

LGTM!

[can-anyscale]

w00t I just rebase your PR on top of the latest master to avoid a bug in microcheck, hope you don't mind, thankks