Support specifying a conda/virtual environment to use in the LocalRun run-config

[jcrist]

When deploying a flow locally using a LocalRun run-config, it would be useful to support running a flow using a different python environment than the one the agent is currently running. This would let users create and deploy flows using isolated environments (i.e. a conda environment per flow).

The types of environments we'd like to support:

• A conda environment, specified either by name or absolute path
• A virtual environment, specified by absolute path
• A python interpreter to use, specified by name on PATH (e.g. python3.8) or full path to interpreter

We could do this with either:

• A single kwarg (python_env?) that takes in a URI where the scheme specifies the type of environment. This is also used by dask-yarn, but may be non-intuitive. Examples:

  • python_env="conda://my-env": a conda environment by name
  • python_env="conda:///full/path/to/env": a conda env by path
  • python_env="venv:///full/path/to/venv": a virtual environment by path
  • python_env="python:///full/path/to/python": full path to a python interpreter

• One kwarg for each type. Specifying multiple kwargs is an error (can't have both a conda and venv). Examples:

  • conda_env="my-env": a conda environment by name
  • conda_env="/full/path/to/env": a conda environment by full path
  • virtual_env="/full/path/to/env": a virtual environment by full path
  • python_env="/full/path/to/python": full path to a python interpreter
  • python_env="python3.8": python interpreter command name

I'm leaning towards the second option, as it seems clearer to me.

[pedrocava]

This would definitely be appreciated by my team

[zanieb]

I concur that the first option is confusing. We could perhaps get away with just python_env and inferring the type of environment by inspecting the file at the given path or giving precedence to python -> conda -> virtualenv for environment names. The explicit option is probably the easiest to use though.

[edemirel]

There's a hacky workaround for this that I just tested. Not sure if it's the right way to do things but technically, if you setup a virtual environment, register the flow from THAT virtual environment and then actually start a specific agent with specific labels from that environment, it should work. Technically, you'd be running two separate local agents on the same computer essentially.

Not syncing the registering environment and the agent environment ends up something like below

Failed to load and execute Flow's environment: FlowStorageError("An error occurred while unpickling the flow:\n TypeError('an integer is required (got type bytes)')\nThis may be due to one of the following version mismatches between the flow build and execution environments:\n - cloudpickle: (flow built with '2.0.0', currently running with '1.6.0')\n - python: (flow built with '3.6.13', currently running with '3.8.8')")

[zanieb]

We're implementing this in Orion, so I'll close this for now.

[Romathonat]

Can we do this so far with Orion ? I did not find info in the doc

[zanieb]

This will be included in the next Orion release :)

[oxydaemon]

I'm interested in this feature as well, but don't see any reference in the changelog yet, just want to double check, by next Orion release with this feature do you mean next major release (i.e. 1.0)? Thanks!

[zanieb]

Orion was our code name for the 2.0 version. This feature is included in 2.0a7. It won't be included in 1.0.