From 16b7740efb412f0b1bf51b0579044b606e08b136 Mon Sep 17 00:00:00 2001 From: Nick Coghlan Date: Sun, 5 Mar 2023 21:24:37 +1000 Subject: [PATCH 1/4] Clarify specs for external installer interactions Closes #1210, #1211, #1212 --- source/specifications/direct-url.rst | 5 +- source/specifications/entry-points.rst | 5 +- .../externally-managed-environments.rst | 22 +++++++ source/specifications/index.rst | 10 +++- .../recording-installed-packages.rst | 59 ++++++++++++++++--- .../specifications/virtual-environments.rst | 40 +++++++++++++ 6 files changed, 129 insertions(+), 12 deletions(-) create mode 100644 source/specifications/externally-managed-environments.rst create mode 100644 source/specifications/virtual-environments.rst diff --git a/source/specifications/direct-url.rst b/source/specifications/direct-url.rst index fc00fa132..7a1560e87 100644 --- a/source/specifications/direct-url.rst +++ b/source/specifications/direct-url.rst @@ -6,8 +6,9 @@ Recording the Direct URL Origin of installed distributions ========================================================== This document specifies a :file:`direct_url.json` file in the -:file:`*.dist-info` directory of an installed distribution, to record the -Direct URL Origin of the distribution. +``*.dist-info`` directory of an installed distribution, to record the +Direct URL Origin of the distribution. The general structure and usage of +``*.dist-info`` directories is described in :ref:`recording-installed-packages`. .. contents:: Contents :local: diff --git a/source/specifications/entry-points.rst b/source/specifications/entry-points.rst index 6c94a25fb..a11f3ca93 100644 --- a/source/specifications/entry-points.rst +++ b/source/specifications/entry-points.rst @@ -81,8 +81,9 @@ File format Entry points are defined in a file called :file:`entry_points.txt` in the :file:`*.dist-info` directory of the distribution. This is the directory -described in :pep:`376` for installed distributions, and in :pep:`427` for -wheels. The file uses the UTF-8 character encoding. +described in :ref:`recording-installed-packages` for installed distributions, +and in :ref:`binary-distribution-format` for wheels. +The file uses the UTF-8 character encoding. The file contents are in INI format, as read by Python's :mod:`configparser` module. However, configparser treats names as case-insensitive by default, diff --git a/source/specifications/externally-managed-environments.rst b/source/specifications/externally-managed-environments.rst new file mode 100644 index 000000000..dff0909ec --- /dev/null +++ b/source/specifications/externally-managed-environments.rst @@ -0,0 +1,22 @@ + +.. _externally-managed-environments: + +=============================== +Externally Managed Environments +=============================== + +While some Python installations are entirely managed by the user that installed +Python, others may be provided and managed by another means (such as the +operating system package manager in a Linux distribution, or as a bundled +Python environment in an application in a dedicated installer). + +Attempting to use conventional Python packaging tools to manipulate such +environments can be confusing at best and outright break the entire underlying +operating system at worst. Documentation and interoperability guides only go +so far in resolving such problems. + +:pep:`668` defined an ``EXTERNALLY-MANAGED`` marker file that allows a Python +installation to indicate to Python-specific tools such as ``pip`` that they +neither install nor remove packages into the interpreter’s default installation +environment, and should instead guide the end user towards using +:ref:`virtual-environments`. diff --git a/source/specifications/index.rst b/source/specifications/index.rst index 3265ce56f..f4a290e29 100644 --- a/source/specifications/index.rst +++ b/source/specifications/index.rst @@ -22,10 +22,18 @@ Package Distribution Metadata declaring-build-dependencies declaring-project-metadata platform-compatibility-tags + +Package Installation Environment Metadata +----------------------------------------- + +.. toctree:: + :maxdepth: 1 + recording-installed-packages entry-points direct-url - + virtual-environments + externally-managed-environments Package Distribution File Formats --------------------------------- diff --git a/source/specifications/recording-installed-packages.rst b/source/specifications/recording-installed-packages.rst index 6a01544d5..f354fc52e 100644 --- a/source/specifications/recording-installed-packages.rst +++ b/source/specifications/recording-installed-packages.rst @@ -21,12 +21,14 @@ History and change workflow =========================== The metadata described here was first specified in :pep:`376`, and later -amended in :pep:`627`. +amended in :pep:`627` (and other PEPs). It was formerly known as *Database of Installed Python Distributions*. -Further amendments (except trivial language or typography fixes) must be made -through the PEP process (see :pep:`1`). +As with other PyPA specifications, editorial amendments with no functional +impact may be made through the GitHub pull request workflow. Proposals for +functional changes that would require amendments to package building and/or +installation tools must be made through the PEP process (see :pep:`1`). -While this document is the normative specification, these PEPs that introduce +While this document is the normative specification, the PEPs that introduce changes to it may include additional information such as rationales and backwards compatibility considerations. @@ -66,12 +68,14 @@ encouraged to start normalizing those fields. provide API for such tools to consume, so tools can have access to the unnormalized name when displaying distrubution information. -This ``.dist-info`` directory can contain these files, described in detail -below: +This ``.dist-info`` directory may contain the following files, described in +detail below: * ``METADATA``: contains project metadata * ``RECORD``: records the list of installed files. * ``INSTALLER``: records the name of the tool used to install the project. +* ``entry_points.txt``: see :ref:`entry-points` for details +* ``direct_url.json``: see :ref:`direct-url` for details The ``METADATA`` file is mandatory. All other files may be omitted at the installing tool's discretion. @@ -169,7 +173,7 @@ Here is an example snippet of a possible ``RECORD`` file:: If the ``RECORD`` file is missing, tools that rely on ``.dist-info`` must not attempt to uninstall or upgrade the package. -(This does not apply to tools that rely on other sources of information, +(This restriction does not apply to tools that rely on other sources of information, such as system package managers in Linux distros.) @@ -197,6 +201,18 @@ For example, if a tool is asked to uninstall a project but finds no ``RECORD`` file, it may suggest that the tool named in ``INSTALLER`` may be able to do the uninstallation. + +The entry_points.txt file +========================= + +This file MAY be created by installers to indicate when packages contain +components intended or discovery and use by other code, including console +scripts and other applications that the installer has made available for +execution. + +Its detailed specification is at :ref:`entry-points`. + + The direct_url.json file ======================== @@ -207,3 +223,32 @@ This file MUST NOT be created when installing a distribution from an other type of requirement (i.e. name plus version specifier). Its detailed specification is at :ref:`direct-url`. + + +Intentionally preventing changes to installed packages +====================================================== + +In some cases (such as when needing to manage external dependencies in addition +to Python ecosystem dependencies), it is desirable for a tool that installs +packages into a Python environment to ensure that other tools are not used to +uninstall or otherwise modify that installed package, as doing so may cause +compatibility problems with the wider environment. + +To achieve this, affected tools should take the folllowing steps: + +* Rename or remove the ``RECORD`` file to prevent changes via other tools (e.g. + appending a suffix to create a non-standard ``RECORD.tool`` file if the tool + itself needs the information, or omitting the file entirely if the package + contents are tracked and managed via other means) +* Write an ``INSTALLER`` file indicating the name of the tool that should be used + to manage the package (this allows ``RECORD``-aware tools to provide better + error notices when asked to modify affected packages) + +Python runtime providers may also prevent inadvertent modification of platform +provided packages by modifying the default Python package installation scheme +to use a location other than that used by platform provided packages (while also +ensuring both locations appear on the default Python import path). + +In some circumstances, it may be desirable to block even installation of +additional packages via Python-specific tools. For these cases refer to +:ref:`externally-managed-environments` \ No newline at end of file diff --git a/source/specifications/virtual-environments.rst b/source/specifications/virtual-environments.rst new file mode 100644 index 000000000..9c0dba674 --- /dev/null +++ b/source/specifications/virtual-environments.rst @@ -0,0 +1,40 @@ + +.. _virtual-environments: + +=========================== +Python Virtual Environments +=========================== + +In Python 3.3 and later versions, :pep:`405` provides interpreter level support +for the concept of "Python Virtual Environments". Each virtual environment has +its own Python binary (allowing creation of environments with various Python +versions) and can have its own independent set of installed Python packages in +its site directories, but shares the standard library with the base installed +Python. While the concept of virtual environments existed prior to this update, +there was no previously standardised mechanism for declaring or discovering them. + +Runtime detection of virtual environments +========================================= + +At runtime, virtual environments can be identified by virtue of ``sys.prefix`` +(the filesystem location of the running interpreter) having a different value +from ``sys.base_prefix`` (the default filesytem location of the standard library +directories). + +Declaring installation environments as Python virtual environments +================================================================== + +As described in :pep:`405`, a Python virtual environment in its simplest form +consists of nothing more than a copy or symlink of the Python binary accompanied +by a ``site-packages`` directory and a ``pyvenv.cfg`` file with a ``home`` key +that indicates where to find the Python standard library modules. + +This split installation and ``pyvenv.cfg`` file approach can be used by *any* +Python installation provider that desires Python-specific tools to be aware that +they are already operating in a virtual environment and no further environment +nesting is required or desired. + +However, any approach that results in ``sys.prefix`` and ``sys.base_prefix`` +having different values, while still providing a valid package installation +scheme in ``sysconfig``, will be detected and behave as a Python virtual +environment. From 78f18f3d45d841cd1fd9e919690f0915a44f35d9 Mon Sep 17 00:00:00 2001 From: Nick Coghlan Date: Sun, 5 Mar 2023 21:28:37 +1000 Subject: [PATCH 2/4] Working tweak --- source/specifications/externally-managed-environments.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/specifications/externally-managed-environments.rst b/source/specifications/externally-managed-environments.rst index dff0909ec..3876d8e34 100644 --- a/source/specifications/externally-managed-environments.rst +++ b/source/specifications/externally-managed-environments.rst @@ -8,7 +8,7 @@ Externally Managed Environments While some Python installations are entirely managed by the user that installed Python, others may be provided and managed by another means (such as the operating system package manager in a Linux distribution, or as a bundled -Python environment in an application in a dedicated installer). +Python environment in an application with a dedicated installer). Attempting to use conventional Python packaging tools to manipulate such environments can be confusing at best and outright break the entire underlying From 3b82319464d58cc7196863716311c9ce5060e838 Mon Sep 17 00:00:00 2001 From: Nick Coghlan Date: Sun, 5 Mar 2023 23:51:14 +1000 Subject: [PATCH 3/4] Update source/specifications/recording-installed-packages.rst Co-authored-by: Paul Moore --- source/specifications/recording-installed-packages.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/specifications/recording-installed-packages.rst b/source/specifications/recording-installed-packages.rst index f354fc52e..bac887b2b 100644 --- a/source/specifications/recording-installed-packages.rst +++ b/source/specifications/recording-installed-packages.rst @@ -206,7 +206,7 @@ The entry_points.txt file ========================= This file MAY be created by installers to indicate when packages contain -components intended or discovery and use by other code, including console +components intended for discovery and use by other code, including console scripts and other applications that the installer has made available for execution. From 88f0c1af99b6c331b8a161fb6edd5c90038126ef Mon Sep 17 00:00:00 2001 From: Nick Coghlan Date: Mon, 6 Mar 2023 00:17:22 +1000 Subject: [PATCH 4/4] Reference main venv docs --- .../specifications/virtual-environments.rst | 22 ++++++++++++++----- 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/source/specifications/virtual-environments.rst b/source/specifications/virtual-environments.rst index 9c0dba674..c38c2faeb 100644 --- a/source/specifications/virtual-environments.rst +++ b/source/specifications/virtual-environments.rst @@ -5,7 +5,7 @@ Python Virtual Environments =========================== -In Python 3.3 and later versions, :pep:`405` provides interpreter level support +For Python 3.3 and later versions, :pep:`405` introduced interpreter level support for the concept of "Python Virtual Environments". Each virtual environment has its own Python binary (allowing creation of environments with various Python versions) and can have its own independent set of installed Python packages in @@ -13,6 +13,7 @@ its site directories, but shares the standard library with the base installed Python. While the concept of virtual environments existed prior to this update, there was no previously standardised mechanism for declaring or discovering them. + Runtime detection of virtual environments ========================================= @@ -21,6 +22,13 @@ At runtime, virtual environments can be identified by virtue of ``sys.prefix`` from ``sys.base_prefix`` (the default filesytem location of the standard library directories). +:ref:`venv-explanation` in the Python standard library documentation for the +:py:mod:`venv` module covers this along with the concept of "activating" a +virtual environment in an interactive operating system shell (this activation +step is optional and hence the changes it makes can't be reliably used to +detect whether a Python program is running in a virtual environment or not). + + Declaring installation environments as Python virtual environments ================================================================== @@ -29,12 +37,14 @@ consists of nothing more than a copy or symlink of the Python binary accompanied by a ``site-packages`` directory and a ``pyvenv.cfg`` file with a ``home`` key that indicates where to find the Python standard library modules. -This split installation and ``pyvenv.cfg`` file approach can be used by *any* +While designed to meet the needs of the standard :py:mod:`venv` module, this +split installation and ``pyvenv.cfg`` file approach can be used by *any* Python installation provider that desires Python-specific tools to be aware that they are already operating in a virtual environment and no further environment nesting is required or desired. -However, any approach that results in ``sys.prefix`` and ``sys.base_prefix`` -having different values, while still providing a valid package installation -scheme in ``sysconfig``, will be detected and behave as a Python virtual -environment. +Even in the absence of a ``pyvenv.cfg`` file, any approach (e.g. +``sitecustomize.py``, patching the installed Python runtime) that results in +``sys.prefix`` and ``sys.base_prefix`` having different values, while still +providing a matching default package installation scheme in ``sysconfig``, will +be detected and behave as a Python virtual environment.