type checker project boundaries

[kinto0]

Hey, thanks for working on this: I'm excited about this extension and how it can help in one of the most confusing parts of setting up a Python project in VSCode. I work on Pyrefly and have a few questions about project boundaries as they relate to workspace folders.

I'm sorry I haven't been able to test/try this myself since I don't see the Python tab on the left from the video (I reloaded my window + opened a python file after switching to the pre-release version). Please let me know if I'm missing something there and should test this all myself and re-ask these questions.

In a normal python project, there might be multiple python roots in a project. For example, I might run python3 script.py from root/scripts and use a conda environment for other code execution. IDE services usually work in root/scripts because of Pyright's fallback heuristic of "pretend every parent directory is a project root". This small example is usually OK enough not to create a new environment for each one.

But in a megarepo, this problem becomes much more difficult: there can be thousands of projects with their own python environment. The general advice is to open each project as a workspace folder in VSCode: this lets you use the preexisting interpreter selector to select environments per-project. But this approach does not scale well and many people forget. So in big megarepos, we've used the VSCode API to automatically open any parent workspace folders that have .vscode/settings.json in order to find the defaultInterpreterPath setting and work automatically (it's frustrating that it doesn't upwards-search for this file). This is not great since people don't like us messing with their VSCode settings.

I'm hoping the environments extension can help in the Megarepo since I think the problem is more common than just at Meta.

From reading the documentation on this (sorry I can't test it successfully), I see pythonProjects which is "A list of Python workspaces, specified by the path, in which you can set particular environment and package managers". I wonder if this setting can help us by dynamically adding these for each project root.

• How will these interact with workspace folders? Will it always be 1:1?
• Are there plans for these to be able to find pyproject.tomls that aren't directly at a root workspace folder?

Type checkers like Pyrefly, Ty, Mypy, etc all use a mix of pyproject.tomls, their own configuration, VSCode workspace / environment settings, to understand the "scope" of a project. I wonder if we could allow these extensions to add pythonProjects through some API if they find a configuration file. These checkers (and Pyright does this too) already need upwards-search and create their own combined project scope based on this combined configuration - should they be able to modify the vscode-python-environments project scope?

Thank you!!

[eleanorjboyd]

Hi Kyle! Great to hear about your specific use case at Meta and for Pyrefly users. Our goal with this extension was to make monorepos with many environments easier to manage so hopefully we can find a way for this extension and its tooling to help your problem.

We are still rolling this out to users under experimentation which is likely why you cant see the python icon. If you add python.useEnvironmentsExtension to your user settings it should appear (but let me know if not!)

You are right about projects, our goal there was to connect files/folders to the correct virtual environment to use; allowing users to configure which environment to use for any of their folders. Since we correlate paths (folders or files) with venvs, we can then switch dynamically between venvs in a single workspace depending on what file the user is looking at so you shouldn't need to open each different project as a different workspace folder.

How the projects actually work, which I think gets at your question of how your team can use them, is they are setup by the user and saved to settings. Once you add a path as a project it appears like this in your settings:

"python-envs.pythonProjects": [
        {
            "path": "src/client",
            "envManager": "ms-python.python:venv",
            "packageManager": "ms-python.python:pip"
        }
    ]

Then when you open the file again, we get the environment from the envManager to then use.

It does mean you can define projects in the settings.json that will then be partitioned for anyone on opening the repository in VS Code. Since we don't save the path to the interpreter in the settings, it finds the actual environment by either caching which one the user selected for a specific project or searching within the project path (like src/client in this example) for a .venv (like src/client/.venv) or equivalent prioritization based on the package manager.

You add a path as project by right clicking on that file/folder in the file explorer like this:

The other way, is to click this "add folder" icon in the projects menu on the python sidebar:

and select auto-find from the menu that appears (which will do this search for pyproject.toml like you were talking about)

In summary, I think the envs extension has the tools and parts of what you are looking for and we just have to figure out how to make it work best for your scenarios. Since that setting I wrote should allow you to test it, you can play around with it on you own and see what you think. We can then discuss further about options. Thanks!

[kinto0]

Thanks for the explanation!! The setting works - the UX is really nice and I think it will solve a lot of pain-points our developers have had.

The piece I didn't understand originally was the ability for other extensions to modify these projects. But looking at the api (assuming I'm understanding it correctly), it seems like our extension will have this ability.

I'm imagining a few scenarios:

megarepo workspace folder with isolated project files open

Some projects are executed with their own python runtime defined in the project. they only exist in the megarepo for organization and don't depend on other projects. in these case, we'd want to allow the user to modify their python environment on a per-project basis. the pyrefly extension would be the source of truth for which projects these are (based on a pyrefly configuration in project1/).

workspace folder at /path/to/megarepo
pyrefly configuration at /path/to/megarepo/project1

upon project file open, the pyrefly extension would know project1 is the project boundaries and call addPythonProject(name=project1, uri=/path/to/megarepo/project1)

project workspace folder in megarepo

if a project is tightly tied to the megarepo or a parent folder of a workspace, it should be configurable by the megarepo folder, even if that folder is not directly open in vscode.

workspace folder at /path/to/megarepo/project2
pyrefly configuration at /path/to/megarepo

upon project file open, even though the workspace folder is smaller than the project boundaries, pyrefly would want to addPythonProject(name=megarepo, uri=/path/to/megarepo)

it sounds like these scenarios are part of the plan. I'm excited about migrating to this extension and wish you luck with the rollout!

[eleanorjboyd]

Closing this as the specific question was answered (I think?) but feel free to comment or open a new issue to continue discussion. Thanks!