[environments] Support PEX executables

[edoakes]

This is a feature request for supporting PEX in the runtime_env of ray, which is similar to how conda is supported (see issue #14524 2).

For example, we can package an actor “foo” in a PEX file “my_package.pex”:

@ray.remote class Foo(object): ...

And then initialize ray with the PEX environment to enable the worker to run “foo” in the PEX environment:

ray.init(runtime_env = { "pex": "path/to/my_package.pex" });

We have an example repo GitHub - leozc/fp20 1 to package ray actors into PEX files and run the actor in a local ray cluster. The commands are:

build ray_demo.pex in dist:

./pants package ray_demo:ray_demo

run ray workers in PEX environment:

./dist/ray_demo/ray_demo.pex

What we would like to have is to execute the code in a real cluster (e.g. cloud, k8s, etc.).

[ericl]

Long term it seems likely we're going to have a long tail of extra features for runtime_env. One way of supporting this to make it easy to add new keys to runtime_env. This would involve at least:

1. The ability to easily add new resource types in runtime_env (e.g., updating a dict in runtime_env.py similar to how we have a dict of node providers for autoscaler).
2. A storage API that plugins can use to save data (e.g., the PEX file) for distribution to the cluster.

cc @iycheng

[fishbone]

Long term it seems likely we're going to have a long tail of extra features for runtime_env. One way of supporting this to make it easy to add new keys to runtime_env. This would involve at least:

1. The ability to easily add new resource types in runtime_env (e.g., updating a dict in runtime_env.py similar to how we have a dict of node providers for autoscaler).
2. A storage API that plugins can use to save data (e.g., the PEX file) for distribution to the cluster.

cc @iycheng

Yes, this is what I'm currently working on. Hopefully, I'll get the PR out soon.

[mingshi-wang]

@iycheng Hi Yi, if you could share some pointers on the approach we would be happy to work on this and contribute back!

[mingshi-wang]

@iycheng Hi Yi, I wonder if you are open for a discussion on this feature request?

[fishbone]

Hi @mingshi-wang sorry, I missed your message. Yes, I'm ok with some discussion of this one. I think right now, @architkulkarni also has some thoughts about this one, please involve him as well.

[yehoshoua]

PEX support should be very usefully. i see this request open 6 months ago did you have any estimations ?

[yijianli-autra]

any updates on this? or can we contribute to this feature?

[architkulkarni]

We don't have a timeline for adding this, but it should be possible to write your own "third-party runtime env plugin" for this. We don't have documentation for the plugin API at the moment and it is still experimental, but the following test https://github.com/ray-project/ray/blob/2bedf2490d7b11c2728c3fc8c57cf337a5e296ef/python/ray/tests/test_runtime_env_plugin.py has some example plugins.

If the plugin has enough interest we can consider merging it into the Ray repo. cc @scv119