[Train/Tune] Restore an experiment from a different machine/path

[justinvyu]

What happened + What you expected to happen

1. Run an experiment on machine A
2. Copy the contents of the experiment directory to machine B -- where the path is now different
3. Try to restore for further training/analysis
4. This will error due to absolute paths being saved in the checkpoints

See user issues:

• https://discuss.ray.io/t/restoring-agent-with-ray-tune/12525
• https://discuss.ray.io/t/loading-experiment-analysis-from-a-different-machine-than-the-experiment-was-run-with/12397

Versions / Dependencies

2.7.1

Reproduction script

Same as this issue: #28082

This is a regression introduced in 2.7.0.

Temporary workaround is to make sure the paths on the new machine match up, or use ray<2.7.

Issue Severity

Medium: It is a significant difficulty but I can work around it.