[air] pyarrow.fs persistence (9/n): ray.train.Checkpoint restore: Manual restore

[justinvyu]

Why are these changes needed?

This PR adds support for the new persistence path for manual trainer restoration.

API changes:

Tuner.restore(..., storage_filesystem: Optional[pyarrow.fs.FileSystem] = None)
Tuner.can_restore(..., storage_filesystem: Optional[pyarrow.fs.FileSystem] = None)

BaseTrainer.restore(..., storage_filesystem: Optional[pyarrow.fs.FileSystem] = None)
BaseTrainer.can_restore(..., storage_filesystem: Optional[pyarrow.fs.FileSystem] = None)

This is required to download the pkl files containing the info needed to manually restore an experiment, in the case that the user originally provided a custom fs.

Follow-ups needed

• Tuner restore doesn't work because Tune trainables aren't supported in the new codepath yet.
  • This is being tackled by [air] pyarrow.fs persistence (10/n): Unify Tune and Train sessions to support new persistence path in FunctionTrainable #38284
• This PR does not yet support manual restore from a moved storage path.
  • See these PRs for context:
    • [Tune] Tuner.restore from a local directory that has moved #29920
    • [Tune] Enable experiment restore from moved cloud uri #31669
  • These are labeled with # TODO(justinvyu): [handle_moved_storage_path]
• Update docs

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[justinvyu]

@ericl Ok to merge