Control inclusion of pip and setuptools in virtual environments via config options

[abn]

With the changes introduced in #2826, new virtual environments created will not include pip nor setuptools as poetry no longer assumes them to be present. You can try this by using the version of poetry on master.

While this has many advantages (see #1584), it also has the disadvantage that majority of developer tools (PyCharm, VSCode etc.) out there makes the assumption that pip, pkg_resources and setuptools exists in the environment. Personally, I think this assumption in itself is dangerous. Despite what I think about it, it is the common case and hence we should talk about the impact.

As poetry users are largely using it for development, this is an issue. In order mitigate this concern, we could do the following.

1. Create new config options virtualenvs.options.with_pip and virtualenvs.options.with_setuptools both defaulting to true to keep existing behaviour and perhaps flip it in a couple of releases.
2. Poetry internal operations continue to use pip embedded in virtualenv, thus keeping it out of the unsafe packages list.

The side-effects here (when options are enabled) are;

1. We loose the "bare" virtual environment by default introduced via Drop poetry's dependency on pip and setuptools in project virtual environment #2826.
2. --remove-untracked will remove pip and related packages. (We can potentially try avoiding this with the help of the values configured for the options above).

Another option is to take the stance that if your development environment requires any of these packages, you explicitly add them to your project dependencies.

I would like to hear from the community;

1. If we should add these configurations at all.
2. If we shoud, what should be the default?

The workaround I follow for my development environment is to use a second virtual environment with just IDE requirements and the site-packages added to the project interpretor's paths.

WIP: #3920

[TBBle]

Would it make sense, perhaps controlled by config option, for new projects to include pip and setuptools as dev-dependencies, so that IDEs that expect them to be present are supported out-of-the-box, and then users are free to remove them if not needed?

I'm not sure if and how we could handle this behaviour for existing projects... Perhaps if the mooted virtualenvs.options.with_pip config option is on, Poetry can complain when a project does not contain pip as a dev-dependency, to prompt users to make their projects ready-to-run with that option disabled? (Or make that option have 'on','off', and 'warn' states?)

[apollo13]

I like that, if that works out I'd prefer it over my suggested approach. That said it still might require a bit of explaining for people upgrading from older poetry…

[abn]

The issue here is that one developer might need it and another might not. So putting them into the pyproject as a default feels a bit overkill. And for pycharm, as an example, all it needs is pkg_resources it'd seem.

[TBBle]

"One developer might need them and another might not" is probably counter to the general Poetry intention of "Everyone gets the same thing, locked down in the repo".

I'd certainly rather have something needed by a teammate's IDE included in the dev-dependencies, even if I never use it, than have our environments differ, particularly if the extra package has knock-on consequences, e.g., changes the resolution of other parts of the system, causing tests written with the IDE to fail on my IDE-less setup.

We wouldn't exclude tox from our dev-dependencies just because some developers don't use it to run their tests.

[abn]

That is a slippery slope I feel. Within an organisation "team-mate" is a well defined, within a community project not so much. I'd argue that tox is more project specific than developer specific. Additionally, the problem here is also that not all IDEs or tools make the same assumptions as to what is available in the virtual environment.

Regarding the "knock-on consequences", my personal bias is to say that it is why CI exists. That said, once we have dependency groups this might be slightly better and more feasible.

[TBBle]

I note that VS Code's Python documentation countenances that Linux Python 3 installs may not have pip, and specifies to install it, so this could be documented in the short-term.

They're explicitly adding Poetry support right now (microsoft/vscode-python#8372) and have future plans to support not relying on pip for package management (microsoft/vscode-python#11739, assuming the implementation of that for Poetry envs is poetry add --dev).

And in the meantime, adding a warning in VSCode when a Poetry venv is activated, but pip is not present, should not be too hard, if that doesn't exist already for general venv support in VSCode.

Edit: I looked, and it appears that if a Poetry env is activated, poetry add --dev is used to install VSCode tools already. This version isn't released yet though, it's only out on Insiders per microsoft/vscode-python#8372 (comment). The only other pip-usage I could see outside the PipEnv or Pip environment support was in telemetry, and that's been removed as well: microsoft/vscode-python@1cd761c.

[abn]

This is good to see. Do you know if vscode makes any assumption of the environment when, for example, test suites are run or python console is launched. I'd say the dependency management bit is okay as for poetry managed projects we implicitly expect these to be managed by poetry anyway.

[TBBle]

I haven't played with test discovery or the console much in VSCode (I usually run those in Windows Terminal session on another screen) but from what I've seen with the current behaviour, it starts terminal sessions with the current venv activated, and that works for Poetry venvs now, if you manually select them. So I expect that should all work, as by that point VSCode doesn't care where the venv came from, as long as it's activatable.

[abn]

I did track down what is happening in PyCharm. This is caused by their entrypoint discovery mechanism used by the pytest helper plugins.

[TBBle]

python-poetry/poetry-core#168 (comment) suggests a min-poetry config value, perhaps this could be the hook for "don't install pip and setuptools in the venv". Then the user is actively and in-project changing that behaviour.

My concern with a config option is that different users may end up with different values, and hence see breakages that are not immediately reproducible on CI or other machines.

In this case, I expect that if you take an existing project, change the min-poetry value to 1.2.0, and poetry install --remove-untracked, then pip and setuptools and their unlocked requirements would be uninstalled from the venv.

Of course, the problem with a min-poetry value is that older versions won't support it, and won't know they're too old...

[apollo13]

Personally I think that poetry should just create a normal venv (with pip and setuptools) but always use it's embedded pip for install & removal. If a user then wishes to lock setuptools & pip to a specific version, they can do so… Even if they manage to break their venv, poetry would be able to fix it due to it's usage of the embedded pip. remove-untracked should probably ignore pip/setuptools though and consider them as part of the venv.

I guess there is just to much code out there that makes assumptions about what is present in a venv that this might cause more problems than it is worth in the longrun. That said (especially for deployment) it would be great to still be able to create a venv without pip/setuptools

EDIT:// As for your suggestion of

The workaround I follow for my development environment is to use a second virtual environment with just IDE requirements and the site-packages added to the project interpretor's paths.

This is highly technical and not something I'd want to have to recommend to people. Especially not if other tools have "simpler" integrations.

[abn]

I agree that it's too technical. I was not suggesting it as an official recommendation :).

I guess opt-in config is better for deployment cases and is something we can recommend/document.

[sinoroc]

1. Create new config options virtualenvs.options.with_pip and virtualenvs.options.with_setuptools both defaulting to true to keep existing behaviour and perhaps flip it in a couple of releases.
2. Poetry internal operations continue to use pip embedded in virtualenv, thus keeping it out of the unsafe packages list.

Yes and yes.

Add the options to create virtual environments without pip and/or without setuptools now. Leave them by default for now, and in a later major release default to create virtual environments without them. Projects should declare explicit dependencies (dev or not) on pip and/or setuptools if they need it.

I also see it as a very good thing if poetry does not use pip at all, or if pip usage is really needed poetry uses an "out-of-env" pip (virtualenv's embedded pip wheel seems like a good choice).

---

Related:

In the PyPA community there is some motivation towards making the packaging-related tools work "out-of-env", see this discussion.

Pip does not have any dependencies of its own (they are "vendored" instead), so pip's presence in the virtual environment has very little influence on the dependency resolution and the possibility of conflicting dependencies. Same for setuptools. But I see it as a good practice to try to remove as much as possible from the environment so that only the actual application (or library) and its dependencies are present. Meaning we should also aim to remove all "dev" dependencies (linters, code beautifiers, testers, etc.) from the environment.

[apollo13]

I wouldn't necessarily say it is overkill. Currently it is what everyone creating a venv usually expects. I am not saying that this is good but it is the status quo. Working against that might not be beneficial for poetry in the short run -- the situation might change if packaging changes a bit further but we are not there yet. As for adding it to dev dependencies vs config I'd say that it is more portable if it doesn't require configuration. If a project doesn't need it, it can simply delete it once from project.toml. The config option on the other hand will require people to document it as part of the workflow when installing something via poetry...

…

On Sun, Apr 11, 2021, at 14:32, Arun Babu Neelicattu wrote: The issue here is that one developer might need it and another might not. So putting them into the pyproject as a default feels a bit overkill. And for pycharm, as an example, all it needs is pkg_resources it'd seem. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3916 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAT5CZ4VZBS4ZJBQCYKIE3TIGJG5ANCNFSM42XBIURQ>.

[apollo13]

Sorry, this should have been a reply to #3916 (reply in thread) -- but "reply via email" seems to have it's own idea about "replies". :D

[finswimmer]

I prefer the variant to have an option that, if enabled, will install pip and setuptools within the venv. I guess we should be able to determine if this options are set to true, when running --remove-untracked and exclude them than from the uninstall list.

Furthermore I would prefer to set them to true by default. This let people recognize early, that there is something going on and they can start adopting their workflow or report the issue to their IDE.

[abn]

The reluctance I have for immediately flipping it to true by default is that this will and can break existig workflows. I feel that, if we decide to use true by default it will have to be in 1.4. With 1.2 using false as default and 1.3 raising some warning if it is not configured explicitly.

[TBBle]

I think @finswimmer 's true here means "include pip", not "exclude pip", so it would not break default workflows until it becomes false later.

[abn]

I have added a draft version at #3920, testing would be appreicated.