[PROOF OF CONCEPT] Use venv module to create the base Python where it's present

[pfmoore]

This is not ready for merge - at this point it is simply a proof of concept.

Looking at the stdlib venv module and the virtualenv codebase, it seems that the process of creating the base virtual environment (before we add in all of the scripts and install pip, etc) could be handled using the programming API for venv. There are some differences in API that need to be ironed out, but if it works, this would make virtualenv compatible with venv (in the sense of using the virtual environment support built into core Python).

This isn't any sort of attempt to make the two tools compatible at the UI level, it's purely about using the core mechanism when it's available.

At the moment, all I'm doing here is using the virtualenv CI infrastructure to ensure the change works. It needs a lot more manual testing and work before it's going to be ready to propose for actual merging.

[pfmoore]

Bah. CI uses Python 3.7.2, where venv has a known issue that I haven't decided how to address yet. The other Windows failures may also be venv issues, I didn't do any manual tests with older versions yet as I didn't have them installed. I suspect those are because I'm still installing the modified site.py at this point.

[gaborbernat]

I'm not eager on this, switching to venv makes this tools behaviour Python dependent (venv in 3.5 may well behave differently than 3.7), where the whole point of it is to unify virtualenv behaviour independent of python used, so library users can really on a well-defined behaviour. As such I'm tempted to reject it.

[pfmoore]

No problem, my main reason for experimenting was to see how practical it would be to use venv. I actually think that venv still has some odd edge cases that mean venv-using code will be annoyingly version-specific for a while yet. So I don't think it's something I'd want to take forward anyway. I may still tweak the PR a bit, just to see if I can work out why the tests are failing. I can't do that locally (the tests are taking 30 mins or so on my development PC :-() so I'm cheating by using CI. Hope that's OK.

One thing I will say, though, is that I'd dispute the comment about version dependence. Yes, the internals (of virtualenv and of venv across versions) differ, but the user-facing behaviour (which is fundamentally "give me a Python interpreter that's isolated from my main one") should be the same. At the moment, there are quirks (for example, venv uses sys.base_prefix where virtualenv uses sys.real_prefix) but ideally user code shouldn't have to care which "engine" is implementing the behaviour behind the scenes. And because venv uses support built into the core interpreter, it can be way more robust than virtualenv can ever be.

The problem at the moment is that venv won't stabilise fast enough if people don't use it. And many people use virtual environments through tools like pew or pipenv. Because of API differences, those tools have to choose one or the other, and they choose virtualenv because it's the only option. So we get in a vicious circle of no-one using venv, so venv still has rough edges, which mean people are unwilling to use it...

My thought was that if virtualenv could use the venv "engine", users would get access to the more robust core-based engine, while still having the virtualenv UI and API. But that won't be possible unless this experiment had worked. So I'm now considering trying to write a "wrapper" library that will give a unified API over both venv and virtualenv. I don't know whether higher level tools would pick that up, but at least it will solve the integration problems in a single place. (And yes, the underlying differences will remain, but the wrapper can document and unify a common set of capabilities, and just not support the less universal ones).

[gaborbernat]

While you define the API as the higher level outcome of having a virtualenv, I define the API and goal of virtualenv of stable and unified behavior of the created virtualenvs, hence my stricter requirements. You can use the PR as cheap runner as much as you want 👍

[pfmoore]

stable and unified behavior of the created virtualenvs

I'm not disagreeing with this goal at all, but I'd be curious to know what you class as part of the "behaviour". As an example, do you consider having an identical site.py (even though the stdlib site.py varies in both detail and features by version) as being a key feature? Or the existence of the create_virtualenv function? My concern is that you risk falling into the trap of distutils, where the lack of a clear statement about what was public and what was private meant that everything ended up needing to be supported in perpetuity.

Ideally, I'd like any virtualenv/venv wrapper I might write to preserve as many of the guarantees of virtualenv as I can - but without knowing for sure what those guarantees are, that's going to be hard.

[uranusjr]

Not sure how useful it is, but I wrote a wrapper library on top of virtualenv and venv to abstract virtual environment creation across Python versions. It basically tries to use venv if possible[1], and fallback to virtualenv if that fails.

[1]: The logic is to try importing venv on 3.5 or later. 3.4’s venv is not used due to limitations in the Python API.

https://github.com/sarugaku/virtenv/

[gaborbernat]

I'm not disagreeing with this goal at all, but I'd be curious to know what you class as part of the "behaviour". As an example, do you consider having an identical site.py (even though the stdlib site.py varies in both detail and features by version) as being a key feature?

If that site.py alters the way imports work, or how the virtualenv behaves then yes. It's part of the API. The expectation of where I draw the line is that virtualenv env; env/bin/pip install x; env/bin/python -c 'import x' should succeed all the same, given the packaging/installation uses the same unified code.