[air] Don't set rank-specific local directories for Train workers

[justinvyu]

Why are these changes needed?

This PR is based on the REP here: ray-project/enhancements#35

We don't want to force train workers to have separated working directories (rank_x). Instead, this can be configured by the user themselves.

This auto chdir behavior led to issues in the past, where certain frameworks expect a common working directory for each worker (#35894).

# Before
def train_loop_per_worker():
    assert os.getcwd() == "{storage_path}/{exp_name}/{trial_name}/rank_x

# After
def train_loop_per_worker():
    assert os.getcwd() == "{storage_path}/{exp_name}/{trial_name}

    # Write to a unique rank dir yourself if you want to.
    rank_dir = f"rank_{train.get_context().get_world_rank()}"
    os.makedirs(rank_dir, exist_ok=True)
    with open(os.path.join(rank_dir, "artifact.txt")) as f:
        f.write("asdfsadf")

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[justinvyu]

Note: Ray actors don't inherit the working directory of their parent. Even though the Trainable actor changes its directory by default to the trial dir, the worker actor still starts at the original working dir.

[woshiyyya]

Thank you Justin! It looks good to me. I love this change and it would resolve many inconsistency issues for lightning/deepspeed.