bpo-45413: Define "posix_venv", "nt_venv" and "venv" sysconfig instal…

…lation schemes (GH-31034)

Define *posix_venv* and *nt_venv* sysconfig installation schemes
to be used for bootstrapping new virtual environments.
Add *venv* sysconfig installation scheme to get the appropriate one of the above.
The schemes are identical to the pre-existing
*posix_prefix* and *nt* install schemes.
The venv module now uses the *venv* scheme to create new virtual environments
instead of hardcoding the paths depending only on the platform. Downstream
Python distributors customizing the *posix_prefix* or *nt* install
scheme in a way that is not compatible with the install scheme used in
virtual environments are encouraged not to customize the *venv* schemes.
When Python itself runs in a virtual environment,
sysconfig.get_default_scheme and
sysconfig.get_preferred_scheme with `key="prefix"` returns
*venv*.

Doc/library/sysconfig.rst

@@ -73,7 +73,7 @@ Every new component that is installed using :mod:`distutils` or a
 Distutils-based system will follow the same scheme to copy its file in the right
 places.
 
-Python currently supports six schemes:
+Python currently supports nine schemes:
 
 - *posix_prefix*: scheme for POSIX platforms like Linux or macOS.  This is
   the default scheme used when Python or a component is installed.
@@ -83,8 +83,14 @@ Python currently supports six schemes:
 - *posix_user*: scheme for POSIX platforms used when a component is installed
   through Distutils and the *user* option is used.  This scheme defines paths
   located under the user home directory.
+- *posix_venv*: scheme for :mod:`Python virtual environments <venv>` on POSIX
+  platforms; by default it is the same as *posix_prefix* .
 - *nt*: scheme for NT platforms like Windows.
 - *nt_user*: scheme for NT platforms, when the *user* option is used.
+- *nt_venv*: scheme for :mod:`Python virtual environments <venv>` on NT
+  platforms; by default it is the same as *nt* .
+- *venv*: a scheme with values from ether *posix_venv* or *nt_venv* depending
+  on the platform Python runs on
 - *osx_framework_user*: scheme for macOS, when the *user* option is used.
 
 Each scheme is itself composed of a series of paths and each path has a unique
@@ -119,6 +125,9 @@ identifier.  Python currently uses eight paths:
       This function was previously named ``_get_default_scheme()`` and
       considered an implementation detail.
 
+   .. versionchanged:: 3.11
+      When Python runs from a virtual environment,
+      the *venv* scheme is returned.
 
 .. function:: get_preferred_scheme(key)
 
@@ -132,6 +141,10 @@ identifier.  Python currently uses eight paths:
 
    .. versionadded:: 3.10
 
+   .. versionchanged:: 3.11
+      When Python runs from a virtual environment and ``key="prefix"``,
+      the *venv* scheme is returned.
+
 
 .. function:: _get_preferred_schemes()
 

Doc/library/venv.rst

@@ -177,6 +177,11 @@ creation according to their needs, the :class:`EnvBuilder` class.
         ``clear=True``, contents of the environment directory will be cleared
         and then all necessary subdirectories will be recreated.
 
+        .. versionchanged:: 3.11
+           The *venv*
+           :ref:`sysconfig installation scheme <installation_paths>`
+           is used to construct the paths of the created directories.
+
     .. method:: create_configuration(context)
 
         Creates the ``pyvenv.cfg`` configuration file in the environment.

Doc/whatsnew/3.11.rst

@@ -361,6 +361,24 @@ sys
   (equivalent to ``sys.exc_info()[1]``).
   (Contributed by Irit Katriel in :issue:`46328`.)
 
+
+sysconfig
+---------
+
+* Two new :ref:`installation schemes <installation_paths>`
+  (*posix_venv*, *nt_venv* and *venv*) were added and are used when Python
+  creates new virtual environments or when it is running from a virtual
+  environment.
+  The first two schemes (*posix_venv* and *nt_venv*) are OS-specific
+  for non-Windows and Windows, the *venv* is essentially an alias to one of
+  them according to the OS Python runs on.
+  This is useful for downstream distributors who modify
+  :func:`sysconfig.get_preferred_scheme`.
+  Third party code that creates new virtual environments should use the new
+  *venv* installation scheme to determine the paths, as does :mod:`venv`.
+  (Contributed by Miro Hrončok in :issue:`45413`.)
+
+
 threading
 ---------
 
@@ -395,6 +413,20 @@ unicodedata
 * The Unicode database has been updated to version 14.0.0. (:issue:`45190`).
 
 
+venv
+----
+
+* When new Python virtual environments are created, the *venv*
+  :ref:`sysconfig installation scheme <installation_paths>` is used
+  to determine the paths inside the environment.
+  When Python runs in a virtual environment, the same installation scheme
+  is the default.
+  That means that downstream distributors can change the default sysconfig install
+  scheme without changing behavior of virtual environments.
+  Third party code that also creates new virtual environments should do the same.
+  (Contributed by Miro Hrončok in :issue:`45413`.)
+
+
 fcntl
 -----
 

Lib/sysconfig.py

@@ -56,8 +56,53 @@
         'scripts': '{base}/Scripts',
         'data': '{base}',
         },
+    # Downstream distributors can overwrite the default install scheme.
+    # This is done to support downstream modifications where distributors change
+    # the installation layout (eg. different site-packages directory).
+    # So, distributors will change the default scheme to one that correctly
+    # represents their layout.
+    # This presents an issue for projects/people that need to bootstrap virtual
+    # environments, like virtualenv. As distributors might now be customizing
+    # the default install scheme, there is no guarantee that the information
+    # returned by sysconfig.get_default_scheme/get_paths is correct for
+    # a virtual environment, the only guarantee we have is that it is correct
+    # for the *current* environment. When bootstrapping a virtual environment,
+    # we need to know its layout, so that we can place the files in the
+    # correct locations.
+    # The "*_venv" install scheme is a scheme to bootstrap virtual environments,
+    # essentially identical to the default posix_prefix/nt schemes.
+    # Downstream distributors who patch posix_prefix/nt scheme are encouraged to
+    # leave the following schemes unchanged
+    'posix_venv': {
+        'stdlib': '{installed_base}/{platlibdir}/python{py_version_short}',
+        'platstdlib': '{platbase}/{platlibdir}/python{py_version_short}',
+        'purelib': '{base}/lib/python{py_version_short}/site-packages',
+        'platlib': '{platbase}/{platlibdir}/python{py_version_short}/site-packages',
+        'include':
+            '{installed_base}/include/python{py_version_short}{abiflags}',
+        'platinclude':
+            '{installed_platbase}/include/python{py_version_short}{abiflags}',
+        'scripts': '{base}/bin',
+        'data': '{base}',
+        },
+    'nt_venv': {
+        'stdlib': '{installed_base}/Lib',
+        'platstdlib': '{base}/Lib',
+        'purelib': '{base}/Lib/site-packages',
+        'platlib': '{base}/Lib/site-packages',
+        'include': '{installed_base}/Include',
+        'platinclude': '{installed_base}/Include',
+        'scripts': '{base}/Scripts',
+        'data': '{base}',
+        },
     }
 
+# For the OS-native venv scheme, we essentially provide an alias:
+if os.name == 'nt':
+    _INSTALL_SCHEMES['venv'] = _INSTALL_SCHEMES['nt_venv']
+else:
+    _INSTALL_SCHEMES['venv'] = _INSTALL_SCHEMES['posix_venv']
+
 
 # NOTE: site.py has copy of this function.
 # Sync it when modify this function.
@@ -251,6 +296,8 @@ def _get_preferred_schemes():
 
 
 def get_preferred_scheme(key):
+    if key == 'prefix' and sys.prefix != sys.base_prefix:
+        return 'venv'
     scheme = _get_preferred_schemes()[key]
     if scheme not in _INSTALL_SCHEMES:
         raise ValueError(

Lib/test/test_sysconfig.py

@@ -139,6 +139,72 @@ def test_get_preferred_schemes(self):
         self.assertIsInstance(schemes, dict)
         self.assertEqual(set(schemes), expected_schemes)
 
+    def test_posix_venv_scheme(self):
+        # The following directories were hardcoded in the venv module
+        # before bpo-45413, here we assert the posix_venv scheme does not regress
+        binpath = 'bin'
+        incpath = 'include'
+        libpath = os.path.join('lib',
+                               'python%d.%d' % sys.version_info[:2],
+                               'site-packages')
+
+        # Resolve the paths in prefix
+        binpath = os.path.join(sys.prefix, binpath)
+        incpath = os.path.join(sys.prefix, incpath)
+        libpath = os.path.join(sys.prefix, libpath)
+
+        self.assertEqual(binpath, sysconfig.get_path('scripts', scheme='posix_venv'))
+        self.assertEqual(libpath, sysconfig.get_path('purelib', scheme='posix_venv'))
+
+        # The include directory on POSIX isn't exactly the same as before,
+        # but it is "within"
+        sysconfig_includedir = sysconfig.get_path('include', scheme='posix_venv')
+        self.assertTrue(sysconfig_includedir.startswith(incpath + os.sep))
+
+    def test_nt_venv_scheme(self):
+        # The following directories were hardcoded in the venv module
+        # before bpo-45413, here we assert the posix_venv scheme does not regress
+        binpath = 'Scripts'
+        incpath = 'Include'
+        libpath = os.path.join('Lib', 'site-packages')
+
+        # Resolve the paths in prefix
+        binpath = os.path.join(sys.prefix, binpath)
+        incpath = os.path.join(sys.prefix, incpath)
+        libpath = os.path.join(sys.prefix, libpath)
+
+        self.assertEqual(binpath, sysconfig.get_path('scripts', scheme='nt_venv'))
+        self.assertEqual(incpath, sysconfig.get_path('include', scheme='nt_venv'))
+        self.assertEqual(libpath, sysconfig.get_path('purelib', scheme='nt_venv'))
+
+    def test_venv_scheme(self):
+        if sys.platform == 'win32':
+            self.assertEqual(
+                sysconfig.get_path('scripts', scheme='venv'),
+                sysconfig.get_path('scripts', scheme='nt_venv')
+            )
+            self.assertEqual(
+                sysconfig.get_path('include', scheme='venv'),
+                sysconfig.get_path('include', scheme='nt_venv')
+            )
+            self.assertEqual(
+                sysconfig.get_path('purelib', scheme='venv'),
+                sysconfig.get_path('purelib', scheme='nt_venv')
+            )
+        else:
+            self.assertEqual(
+                sysconfig.get_path('scripts', scheme='venv'),
+                sysconfig.get_path('scripts', scheme='posix_venv')
+            )
+            self.assertEqual(
+                sysconfig.get_path('include', scheme='venv'),
+                sysconfig.get_path('include', scheme='posix_venv')
+            )
+            self.assertEqual(
+                sysconfig.get_path('purelib', scheme='venv'),
+                sysconfig.get_path('purelib', scheme='posix_venv')
+            )
+
     def test_get_config_vars(self):
         cvars = get_config_vars()
         self.assertIsInstance(cvars, dict)
@@ -267,7 +333,7 @@ def test_get_config_h_filename(self):
         self.assertTrue(os.path.isfile(config_h), config_h)
 
     def test_get_scheme_names(self):
-        wanted = ['nt', 'posix_home', 'posix_prefix']
+        wanted = ['nt', 'posix_home', 'posix_prefix', 'posix_venv', 'nt_venv', 'venv']
         if HAS_USER_BASE:
             wanted.extend(['nt_user', 'osx_framework_user', 'posix_user'])
         self.assertEqual(get_scheme_names(), tuple(sorted(wanted)))

Lib/test/test_venv.py

@@ -236,6 +236,20 @@ def test_prefixes(self):
             out, err = check_output(cmd)
             self.assertEqual(out.strip(), expected.encode(), prefix)
 
+    @requireVenvCreate
+    def test_sysconfig_preferred_and_default_scheme(self):
+        """
+        Test that the sysconfig preferred(prefix) and default scheme is venv.
+        """
+        rmtree(self.env_dir)
+        self.run_with_capture(venv.create, self.env_dir)
+        envpy = os.path.join(self.env_dir, self.bindir, self.exe)
+        cmd = [envpy, '-c', None]
+        for call in ('get_preferred_scheme("prefix")', 'get_default_scheme()'):
+            cmd[2] = 'import sysconfig; print(sysconfig.%s)' % call
+            out, err = check_output(cmd)
+            self.assertEqual(out.strip(), b'venv', err)
+
     if sys.platform == 'win32':
         ENV_SUBDIRS = (
             ('Scripts',),

Lib/venv/__init__.py

@@ -93,6 +93,15 @@ def clear_directory(self, path):
             elif os.path.isdir(fn):
                 shutil.rmtree(fn)
 
+    def _venv_path(self, env_dir, name):
+        vars = {
+            'base': env_dir,
+            'platbase': env_dir,
+            'installed_base': env_dir,
+            'installed_platbase': env_dir,
+        }
+        return sysconfig.get_path(name, scheme='venv', vars=vars)
+
     def ensure_directories(self, env_dir):
         """
         Create the directories for the environment.
@@ -120,27 +129,21 @@ def create_if_needed(d):
         context.executable = executable
         context.python_dir = dirname
         context.python_exe = exename
-        if sys.platform == 'win32':
-            binname = 'Scripts'
-            incpath = 'Include'
-            libpath = os.path.join(env_dir, 'Lib', 'site-packages')
-        else:
-            binname = 'bin'
-            incpath = 'include'
-            libpath = os.path.join(env_dir, 'lib',
-                                   'python%d.%d' % sys.version_info[:2],
-                                   'site-packages')
-        context.inc_path = path = os.path.join(env_dir, incpath)
-        create_if_needed(path)
+        binpath = self._venv_path(env_dir, 'scripts')
+        incpath = self._venv_path(env_dir, 'include')
+        libpath = self._venv_path(env_dir, 'purelib')
+
+        context.inc_path = incpath
+        create_if_needed(incpath)
         create_if_needed(libpath)
         # Issue 21197: create lib64 as a symlink to lib on 64-bit non-OS X POSIX
         if ((sys.maxsize > 2**32) and (os.name == 'posix') and
             (sys.platform != 'darwin')):
             link_path = os.path.join(env_dir, 'lib64')
             if not os.path.exists(link_path):   # Issue #21643
                 os.symlink('lib', link_path)
-        context.bin_path = binpath = os.path.join(env_dir, binname)
-        context.bin_name = binname
+        context.bin_path = binpath
+        context.bin_name = os.path.relpath(binpath, env_dir)
         context.env_exe = os.path.join(binpath, exename)
         create_if_needed(binpath)
         # Assign and update the command to use when launching the newly created

Misc/NEWS.d/next/Library/2022-01-31-15-19-38.bpo-45413.1vaS0V.rst

@@ -0,0 +1,15 @@
+Define *posix_venv* and *nt_venv*
+:ref:`sysconfig installation schemes <installation_paths>`
+to be used for bootstrapping new virtual environments.
+Add *venv* sysconfig installation scheme to get the appropriate one of the above.
+The schemes are identical to the pre-existing
+*posix_prefix* and *nt* install schemes.
+The :mod:`venv` module now uses the *venv* scheme to create new virtual environments
+instead of hardcoding the paths depending only on the platform. Downstream
+Python distributors customizing the *posix_prefix* or *nt* install
+scheme in a way that is not compatible with the install scheme used in
+virtual environments are encouraged not to customize the *venv* schemes.
+When Python itself runs in a virtual environment,
+:func:`sysconfig.get_default_scheme` and
+:func:`sysconfig.get_preferred_scheme` with ``key="prefix"`` returns
+*venv*.