[air] pyarrow.fs persistence (6/n): Fix Trial + Experiment paths to use the StorageContext

[justinvyu]

Why are these changes needed?

This PR cleans up the path handling in Experiment and Trial to use the StorageContext, and marks all the legacy attributes that can soon be removed.

This fixes an issue when specifying a custom storage_filesystem:

Before:

RunConfig(storage_path="mock/a/b/c", storage_filesystem=CustomFileSystem())

# Experiment / Trial path handling would expand `mock/a/b/c` -> `{os.getcwd()}/mock/a/b/c`
# Basically, we would be treating this storage path as a relative path in certain cases.

result_grid = tuner.fit()

# these use the incorrect expanded abs path
result_grid.experiment_path   #  -> os.getcwd/mock/a/b/c/exp_name
result_grid[0].path   #  -> os.getcwd/mock/a/b/c/exp_name/trial_name

After:

# Paths stay relative
result_grid.experiment_path   #  -> mock/a/b/c/exp_name
result_grid[0].path   #  -> mock/a/b/c/exp_name/trial_name

Q: If a user passes in a relative directory without a custom storage fs (aka a local path), should we expand it to an absolute path?

Other comments

There are now only 2 path concepts: fs_path vs local_path. storage_prefix contains the stripped URI prefix information that can be used by consumers (ex: Result) in order to reconstruct a URI similar to storage_path.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ericl]

Should rebase to clean up the diff.

[ericl]

Cool, this looks much cleaner.

[justinvyu]

@ericl Tests look ok for merge now