[runtime env] Parse local pip/conda requirements files locally upon task/actor definition

[architkulkarni]

Why are these changes needed?

When using Ray Client, the .remote() function for actors and tasks are run on the local driver and run a second time on the remote Client server.

Previously, runtime env dict validation took place in .remote(). This caused a bug where a local file path was specified inthe "pip" or "conda" field, because when the validation was run in the Client server, the local file wouldn't be present.

This PR fixes this by separating out the pip and conda validation and moving it to the __init__ and .options() methods for actors and tasks. Here when a filename string is given, it is parsed into the internal representation List[str] (for pip) or Dict (for conda). Even though .options() is called a second time on the client server, the validation is a no-op there because the conda and pip fields are already in the internal representation at that point.

Related issue number

Closes #18976

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[architkulkarni]

(EDIT: should be fixed by the next commit) @edoakes A couple issues here.

1. It looks like the client_convert function reruns the entire _remote() function, including the runtime env rewriting part that we just added above the client_convert call. In particular, that rewriting part gets called twice, once locally and once in the client server. Hence we see the following error:

ValueError: pip-install-test
requests==2.24.0
 is not a valid file

because it already got rewritten from filename -> file contents once, and during the second call it tries to interpret the file contents as a filename.

2. Since our changes are still happening in _remote(), it's not clear how to locally test it, since the validation hasn't been moved to the init() or .options() part.

[edoakes]

Looks good to me, hopefully this works :)

[architkulkarni]

This PR unfortunately does not address #19002, which I think probably has an unrelated root cause.