Support venv/venvPath for execution environments

[smackesey]

I am working on porting CI for a large monorepo (Dagster) from mypy to pyright. The Dagster OSS repo contains ~70 packages. Due to conflicting dependencies, not all packages can be installed in the same environment. I would like to use Pyright's Execution Environment feature to provide access to the test environment used for each package-- then a single run of pyright should be able to analyze the entire repo.

Tox is used to build the test environment for each package. The cleanest way to achieve what I'm after would be to configure executionEnvironments to point at the tox-created environments. Given a tox environment named py39 in the package some/package:

### pyproject.toml

[tool.pyright]

executionEnvironments = [
    { root = "some/package", venvPath = "some/package/.tox", "venv" = "py39" }
]

However, venv and venvPath don't work for executionEnvironments.

My current workaround is to use extraPaths to point to site-packages of the tox venv:

executionEnvironments = [
    { root = "some/package", extraPaths =  ["some/package/.tox/py39/lib/python3.9/site-packages"] }
]

This mostly works, but it appears that editable installs into site-packages (most of our tox envs use multiple editable installs) ( site-packages by .egg-link files are not resolved on extraPaths). I'm not sure if this is a bug or intended behavior.

Describe the solution you'd like

Preferably, allow setting of venv/venvPath for executionEnvironments.

Alternatively, resolve .egg-link files on extraPaths.

[erictraut]

The implementation of execution environments in pyright assumes a common python environment. Adding support for different environments per execution environment would completely change the way they work and would require a full re-implementation. That's something we're unlikely to do.

If you want to use different python environments for different directories within your mono repo, you should set up one pyrightconfig.json (or pyproject.toml) configuration per directory and run pyright once for each subproject. This is effectively multi-root workspaces are supported when using pyright or pylance with VS Code.

[smackesey]

If you want to use different python environments for different directories within your mono repo, you should set up one pyrightconfig.json (or pyproject.toml) configuration per directory and run pyright once for each subproject. This is effectively multi-root workspaces are supported when using pyright or pylance with VS Code.

Will this work correctly when running pyright as a language server (or Pylance) and editing multiple packages under the same root? Ideally there is a single instance of the Pylance/Pyright language server running and serving diagnostics, and those diagnostics reflect the test environment for each package. That's how you get to parity between CI and editor diagnostics in a monorepo.

The implementation of execution environments in pyright assumes a common python environment.

Would you consider enhancing the docs here? I find this assumption surprising in light of the way "execution environments" are described:

Pyright allows multiple “execution environments” to be defined for different portions of your source tree. For example, a subtree may be designed to run with different import search paths or a different version of the python interpreter than the rest of the source base.

What else is a python environment than a python interpreter and a set of import search paths? Given that, I am confused as to what it means for different "execution environments" (presumably python execution environments?) to assume a "common python environment".

[erictraut]

When a multi-root workspace is used, pylance instantiates an independent pyright service for each workspace root. Any source file that is common to (or referenced by) multiple workspace roots is analyzed multiple times. This increases the memory and CPU usage because it's effectively running many copies of pyright in parallel, but this is necessary because each workspace root is using independent settings (possibly even different python environments), so type evaluations and other analysis results cannot be shared across workspace roots.

Out of curiosity, how do you use mypy with your mono-repo today? I'm guessing that you invoke mypy once for each subproject. If that's true, then the same approach should work with pyright.

The "execution environment" support within pyright has many limitations and should probably be deprecated. It works OK for mono-repos we have internal to Microsoft, but it works only if certain (rather limiting) assumptions are made. One of those assumptions is that your type checking either depends on no Python environment (i.e. you have local stubs for every dependency) or you have a common Python environment that has the superset of all subproject dependencies installed within it.

[smackesey]

Out of curiosity, how do you use mypy with your mono-repo today? I'm guessing that you invoke mypy once for each subproject. If that's true, then the same approach should work with pyright.

That's correct, although this is one of the things we were hoping to improve upon with a switch to pyright. While there's not much difference between 1 and multiple per-package runs in CI, for analyzing the codebase locally it would be much faster and more convenient to do a single run. But the real prize is the kind of editor<->CI parity I mentioned above, which, if I follow you, is not supported? That is, pyright language server mode/pylance can't be configured for a scenario where there are multiple packages with conflicting environment requirements nested under a single source root?

The "execution environment" support within pyright has many limitations and should probably be deprecated. It works OK for mono-repos we have internal to Microsoft, but it works only if certain (rather limiting) assumptions are made. One of those assumptions is that your type checking either depends on no Python environment (i.e. you have local stubs for every dependency) or you have a common Python environment that has the superset of all subproject dependencies installed within it.

I'm a little confused by this because the scheme I outlined above (using per-package execution environments with extraPaths pointing to <package>/.tox/<venv-name>/site-packages) appears to be working quite well, with the exception of not following egg-link files. That said, I haven't much exercised it yet, so perhaps there are some land mines I'll run into?

[erictraut]

I think you can get parity between the editor and CI if you use multi-root workspaces in the editor and independent runs of pyright in CI (one for each workspace root directory). As I mentioned, this is effectively how pylance implements multi-root workspace support.

The extraPaths setting isn't designed to point to site-packages, so you're likely to hit many problems with this approach. You may be able to get it to work if you make certain assumptions, but if you hit problems, our response is going to be "it wasn't designed for this use, so you're on your own". The egg-link files is probably just the tip of the iceberg here.