Add install scheme for virtual environments #89576
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
assignee = 'https://github.com/FFY00' closed_at = <Date 2022-03-18.16:07:36.702> created_at = <Date 2021-10-08.17:50:02.466> labels = ['type-feature', '3.10', '3.11'] title = 'Add install scheme for virtual environments' updated_at = <Date 2022-03-18.16:07:36.702> user = 'https://github.com/FFY00'
activity = <Date 2022-03-18.16:07:36.702> actor = 'FFY00' assignee = 'FFY00' closed = True closed_date = <Date 2022-03-18.16:07:36.702> closer = 'FFY00' components =  creation = <Date 2021-10-08.17:50:02.466> creator = 'FFY00' dependencies =  files =  hgrepos =  issue_num = 45413 keywords = ['patch'] message_count = 16.0 messages = ['403485', '403486', '403513', '403720', '412201', '412202', '412302', '412794', '412805', '412896', '412911', '412933', '413405', '414510', '415479', '415509'] nosy_count = 8.0 nosy_names = ['jaraco', 'petr.viktorin', 'stefanor', 'steve.dower', 'hroncok', 'frenzy', 'gaborjbernat', 'FFY00'] pr_nums = ['31034'] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'enhancement' url = 'https://bugs.python.org/issue45413' versions = ['Python 3.10', 'Python 3.11']
The text was updated successfully, but these errors were encountered:
Python 3.10 introduced sysconfig._get_preferred_schemes, a mechanism to allow downstream distributors to overwrite the default install scheme.
This presents an issue for projects/people that need to bootstrap virtual environments, like virtualenv (see ). As distributors might now be customizing the default install scheme, there is no guarantee that the information returned by sysconfig.get_default_scheme/get_paths is correct for the virtual environment, the only guarantee we have is that it correct for the *current* environment. When bootstrapping a virtual environment, we need to know its layout, so that we can place the files in the correct locations.
The usual solution in situations like this would be to invoke the interpreter we are targeting to get the correct information (eg. path/to/python -m sysconfig), but that is not possible here as the environment does not exist yet -- we are the ones trying to create it.
To solve this issue, I propose the addition of a "virtual" or "venv" install scheme, for virtual environments. The idea is that virtual environments (defined by the presence of pyvenv.cfg, see ) will always use this scheme, they will use the paths specified by this scheme and sysconfig.get_default_scheme will always return it (_get_preferred_schemes would have no effect here!).
I am not cure if this should be adopted in 3.10 or 3.11, as it effectively makes it impossible to reliably construct virtual environments, and requires workarounds such as , that pretty much implements this proposal with non-standardized downstream patches.
The existing install schemes contain values for all different kinds of OSes, somehow named according to them.
If we introduce a single "virtual"/"venv" scheme, it would need to have different contents on different OSes (e.g. Windows vs POSIX). I don't think that would cause any actual trouble, but it would be somewhat different than all the other schemes.
If we introduce multiple ones (e.g. "posix_venv" and "nt_venv") we would need an additional layer to get the one appropriate for the current platform.
Hence, I think having a single one is more pragmatic.
Why? (This is a real question, I genuinely don't know.)
To put this in context, this has been discussed since October, and there was agreement on how to change it.
How can we ensure the decision won't change again?
All I can say is that I wasn't aware of this discussion, or it blurred into the other discussions and didn't seem to need any attention from me.
I definitely don't want to say that I must be consulted on every sysconfig/site/getpath change (because I don't want to be!), but I hesitate to say that this was the wrong way for it to be caught. It's only "too late" after a release has included it, and up until then it's fine.
So I guess it can be avoided in the future by checking the surrounding code and seeing how it's used? In this case, the schemes are all approximately static (for a given version of Python), and the *selection* of a scheme is based on the platform. The proposed venv scheme itself is based on the platform, while selection is static. That stands out to me as a difference.
I don't think the proposal is incompatible with what I discussed.
I haven't been super clear on my opinions on the implementation, so let me try to clarify them.
So, my proposal would be to define a single static scheme, and changing the interpreter path initialization logic to hardcode its paths when on virtual environments.
Hopefully that clarifies things up a bit. We should sort it out as soon as possible and update the PR, I don't think the PR as-is is the best approach.
What do you think?
pre-commit doesn't work on Debian Testing after python is upgraded to 3.10. It seems that this issue is only resolved in python 3.11  ``` 1. pre-commit/pre-commit#2299 2. pre-commit/pre-commit#2336 3. https://bugs.launchpad.net/ubuntu/+source/python3.10/+bug/1967920 4. python/cpython#89576 ``` Signed-off-by: Gabor Nagy <email@example.com>