Improve docs and behavior of editable installs

changelog.d/3414.change.rst

@@ -0,0 +1,4 @@
+Users can *temporarily* specify an environment variable
+``SETUPTOOLS_ENABLE_FEATURE=legacy-editable`` as a escape hatch for the
+:pep:`660` behavior. This setting is **transitional** and may be removed in the
+future.

changelog.d/3414.doc.rst

@@ -0,0 +1,2 @@
+Updated :doc:`Development Mode </userguide/development_mode>` to reflect on the
+implementation of :pep:`660`.

docs/userguide/development_mode.rst

@@ -1,34 +1,155 @@
-Development Mode
-================
-
-Under normal circumstances, the ``setuptools`` assume that you are going to
-build a distribution of your project, not use it in its "raw" or "unbuilt"
-form.  However, if you were to use the ``setuptools`` to build a distribution,
-you would have to rebuild and reinstall your project every time you made a
-change to it during development.
-
-Another problem that sometimes comes is that you may
-need to do development on two related projects at the same time.  You may need
-to put both projects' packages in the same directory to run them, but need to
-keep them separate for revision control purposes.  How can you do this?
-
-Setuptools allows you to deploy your projects for use in a common directory or
-staging area, but without copying any files.  Thus, you can edit each project's
-code in its checkout directory, and only need to run build commands when you
-change files that need to be compiled or the provided metadata and setuptools configuration.
-
-You can perform a ``pip`` installation passing the ``-e/--editable``
-flag (e.g., ``pip install -e .``). It works very similarly to
-``pip install .``, except that it doesn't actually install anything.
-Instead, it creates a special ``.egg-link`` file in the target directory
-(usually ``site-packages``) that links to your project's source code.
-It may also update an existing ``easy-install.pth`` file
-to include your project's source code, thereby making
-it available on ``sys.path`` for all programs using that Python installation.
-
-You can deploy the same project to multiple staging areas, e.g., if you have
-multiple projects on the same machine that are sharing the same project you're
-doing development work.
+Development Mode (a.k.a. "Editable Installs")
+=============================================
+
+When creating a Python project, developers usually want to implement and test
+changes iteratively, before cutting a release and preparing a distribution archive.
+
+In normal circumstances this can be quite cumbersome and require the developers
+to manipulate the ``PATHONPATH`` environment variable or to continuous re-build
+and re-install the project.
+
+To facilitate iterative exploration and experimentation, setuptools allows
+users to instruct the Python interpreter and its import machinery to load the
+code under development directly from the project folder without having to
+copy the files to a different location in the disk.
+This means that changes in the Python source code can immediately take place
+without requiring a new installation.
+
+You can enter this "development mode" by performing an :doc:`editable installation
+<pip:topics/local-project-installs>` inside of a :term:`virtual environment`,
+using :doc:`pip's <pip:cli/pip_install>` ``-e/--editable`` flag, as shown bellow:
+
+.. code-block:: bash
+
+   $ cd your-python-project
+   $ python -m venv .venv
+   # Activate your environemt with:
+   #      `source .venv/bin/activate` on Unix/macOS
+   # or   `.venv\Scripts\activate` on Windows
+
+   $ pip install --editable .
+
+   # Now you have access to your package
+   # as if it was installed in .venv
+   $ python -c "import your_python_project"
+
+
+An "editable installation" works very similarly to a regular install with
+``pip install .``, except that it only installs your package dependencies,
+metadata and wrappers for :ref:`console and GUI scripts <console-scripts>`.
+Under the hood, setuptools will try to create a special :mod:`.pth file <site>`
+in the target directory (usually ``site-packages``) that extends the
+``PYTHONPATH`` or install a custom :doc:`import hook <python:reference/import>`.
 
 When you're done with a given development task, you can simply uninstall your
 package (as you would normally do with ``pip uninstall <package name>``).
+
+Please note that, by default an editable install will expose at least all the
+files that would be available in a regular installation. However, depending on
+the file and directory organization in your project, it might also expose
+as a side effect files that would not be normally available.
+This is allowed so you can iteratively create new Python modules.
+Please have a look on the following section if you are looking for a different behaviour.
+
+.. admonition:: Virtual Environments
+
+   You can think virtual environments as "isolated Python runtime deployments"
+   that allow users to install different sets of libraries and tools without
+   messing with the global behaviour of the system.
+
+   They are the safest way of testing new projects and can be created easily
+   with the :mod:`venv` module from the standard library.
+
+   Please note however that depending on your operating system or distribution,
+   ``venv`` might not come installed by default with Python. For those cases,
+   you might need to use the OS package manager to install it.
+   For example, in Debian/Ubuntu-based systems you can obtain it via:
+
+   .. code-block:: bash
+
+       sudo apt install python3-venv
+
+   Alternatively, you can also try installing :pypi:`virtualenᴠ`.
+   More information is available on the Python Packaging User Guide on
+   :doc:`PyPUG:guides/installing-using-pip-and-virtual-environments`.
+
+.. note::
+    .. versionchanged:: v63.0.0
+       Editable installation hooks implemented according to :pep:`660`.
+       Support for :pep:`namespace packages <420>` is still **EXPERIMENTAL**.
+
+
+"Strict" editable installs
+--------------------------
+
+When thinking about editable installations, users might have the following
+expectations:
+
+1. It should allow developers to add new files (or split/rename existing ones)
+   and have them automatically exposed.
+2. It should behave as close as possible to a regular installation and help
+   users to detect problems (e.g. new files not being included in the distribution).
+
+Unfortunately these expectations are in conflict with each other.
+To solve this problem ``setuptools`` allows developers to choose a more
+*"strict"* mode for the editable installation. This can be done by passing
+a special *configuration setting* via :pypi:`pip`, as indicated bellow:
+
+.. code-block:: bash
+
+    pip install -e . --config-settings editable_mode=strict
+
+In this mode, new files **won't** be exposed and the editable installs will
+try to mimic as much as possible the behavior of a regular install.
+Under the hood, ``setuptools`` will create a tree of file links in an auxiliary
+directory (``$your_project_dir/build``) and add it to ``PYTHONPATH`` via a
+:mod:`.pth file <site>`. (Please be careful to not delete this repository
+by mistake otherwise your files may stop being accessible).
+
+
+.. note::
+    .. versionadded:: v63.0.0
+       *Strict* mode implemented as **EXPERIMENTAL**.
+
+
+Limitations
+-----------
+
+- The *editable* term is used to refer only to Python modules
+  inside the package directories. Non-Python files, external (data) files,
+  executable script files, binary extensions, headers and metadata may be
+  exposed as a *snapshot* of the version they were at the moment of the
+  installation.
+- Adding new dependencies, entry-points or changing your project's metadata
+  require a fresh "editable" re-installation.
+- Console scripts and GUI scripts **MUST** be specified via :doc:`entry-points
+  </userguide/entry_point>` to work properly.
+- *Strict* editable installs require the file system to support
+  either :wiki:`symbolic <symbolic link>` or :wiki:`hard links <hard link>`.
+- Editable installations may not work with
+  :doc:`namespaces created with pkgutil or pkg_resouces
+  <PyPUG:guides/packaging-namespace-packages>`.
+  Please use :pep:`420`-style implicit namespaces.
+- Support for :pep:`420`-style implicit namespace packages for
+  projects structured using :ref:`flat-layout` is still **experimental**.
+  If you experience problems, you can try converting your package structure
+  to the :ref:`src-layout`.
+
+.. attention::
+   Editable installs are **not a perfect replacement for regular installs**
+   in a test environment. When in doubt, please test your projects as
+   installed via a regular wheel. There are tools in the Python ecosystem,
+   like :pypi:`tox` or :pypi:`nox`, that can help you with that
+   (when used with appropriate configuration).
+
+
+Legacy Behavior
+---------------
+
+If your project is not compatible with the new "editable installs" or you wish
+to use the legacy behavior (that mimics the old and deprecated
+``python setup.py develop`` command), you can set an environment variable:
+
+.. code-block::
+
+   SETUPTOOLS_USE_FEATURE="legacy-editable"

docs/userguide/entry_point.rst

@@ -21,6 +21,8 @@ highlighting tool :pypi:`pygments` allows specifying additional styles
 using the entry point ``pygments.styles``.
 
 
+.. _console-scripts:
+
 Console Scripts
 ===============
 

docs/userguide/extension.rst

@@ -56,8 +56,8 @@ a ``foo`` command, you might add something like this to your project:
     distutils.commands =
          foo = mypackage.some_module:foo
 
-(Assuming, of course, that the ``foo`` class in ``mypackage.some_module`` is
-a ``setuptools.Command`` subclass.)
+Assuming, of course, that the ``foo`` class in ``mypackage.some_module`` is
+a ``setuptools.Command`` subclass (documented bellow).
 
 Once a project containing such entry points has been activated on ``sys.path``,
 (e.g. by running ``pip install``) the command(s) will be available to any
@@ -72,9 +72,21 @@ Custom commands should try to replicate the same overall behavior as the
 original classes, and when possible, even inherit from them.
 
 You should also consider handling exceptions such as ``CompileError``,
-``LinkError``, ``LibError``, among others.  These exceptions are available in
+``LinkError``, ``LibError``, among others. These exceptions are available in
 the ``setuptools.errors`` module.
 
+.. autoclass:: setuptools.Command
+   :members:
+
+
+Supporting sdists and editable installs in ``build`` sub-commands
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``build`` sub-commands (like ``build_py`` and ``build_ext``)
+are encouraged to implement the following protocol:
+
+.. autoclass:: setuptools.command.build.SubCommand
+
 
 Adding Arguments
 ----------------

setuptools/__init__.py

@@ -119,7 +119,7 @@ class Command(_Command):
         Most of the time, each option/attribute/cache should only be set if it does not
         have any value yet (e.g. ``if self.attr is None: self.attr = val``).
 
-    .. method: run(self)
+    .. method:: run(self)
 
         Execute the actions intended by the command.
         (Side effects **SHOULD** only take place when ``run`` is executed,

setuptools/build_meta.py

@@ -56,6 +56,9 @@
            '__legacy__',
            'SetupRequirementsError']
 
+SETUPTOOLS_ENABLE_FEATURES = os.getenv("SETUPTOOLS_ENABLE_FEATURES", "").lower()
+LEGACY_EDITABLE = "legacy-editable" in SETUPTOOLS_ENABLE_FEATURES.replace("_", "-")
+
 
 class SetupRequirementsError(BaseException):
     def __init__(self, specifiers):
@@ -419,27 +422,31 @@ def _get_dist_info_dir(self, metadata_directory: Optional[str]) -> Optional[str]
         assert len(dist_info_candidates) <= 1
         return str(dist_info_candidates[0]) if dist_info_candidates else None
 
-    # PEP660 hooks:
-    # build_editable
-    # get_requires_for_build_editable
-    # prepare_metadata_for_build_editable
-    def build_editable(
-        self, wheel_directory, config_settings=None, metadata_directory=None
-    ):
-        # XXX can or should we hide our editable_wheel command normally?
-        info_dir = self._get_dist_info_dir(metadata_directory)
-        opts = ["--dist-info-dir", info_dir] if info_dir else []
-        cmd = ["editable_wheel", *opts, *self._editable_args(config_settings)]
-        return self._build_with_temp_dir(cmd, ".whl", wheel_directory, config_settings)
-
-    def get_requires_for_build_editable(self, config_settings=None):
-        return self.get_requires_for_build_wheel(config_settings)
-
-    def prepare_metadata_for_build_editable(self, metadata_directory,
-                                            config_settings=None):
-        return self.prepare_metadata_for_build_wheel(
-            metadata_directory, config_settings
-        )
+    if not LEGACY_EDITABLE:
+
+        # PEP660 hooks:
+        # build_editable
+        # get_requires_for_build_editable
+        # prepare_metadata_for_build_editable
+        def build_editable(
+            self, wheel_directory, config_settings=None, metadata_directory=None
+        ):
+            # XXX can or should we hide our editable_wheel command normally?
+            info_dir = self._get_dist_info_dir(metadata_directory)
+            opts = ["--dist-info-dir", info_dir] if info_dir else []
+            cmd = ["editable_wheel", *opts, *self._editable_args(config_settings)]
+            return self._build_with_temp_dir(
+                cmd, ".whl", wheel_directory, config_settings
+            )
+
+        def get_requires_for_build_editable(self, config_settings=None):
+            return self.get_requires_for_build_wheel(config_settings)
+
+        def prepare_metadata_for_build_editable(self, metadata_directory,
+                                                config_settings=None):
+            return self.prepare_metadata_for_build_wheel(
+                metadata_directory, config_settings
+            )
 
 
 class _BuildMetaLegacyBackend(_BuildMetaBackend):
@@ -487,12 +494,14 @@ def run_setup(self, setup_script='setup.py'):
 
 get_requires_for_build_wheel = _BACKEND.get_requires_for_build_wheel
 get_requires_for_build_sdist = _BACKEND.get_requires_for_build_sdist
-get_requires_for_build_editable = _BACKEND.get_requires_for_build_editable
 prepare_metadata_for_build_wheel = _BACKEND.prepare_metadata_for_build_wheel
-prepare_metadata_for_build_editable = _BACKEND.prepare_metadata_for_build_editable
 build_wheel = _BACKEND.build_wheel
 build_sdist = _BACKEND.build_sdist
-build_editable = _BACKEND.build_editable
+
+if not LEGACY_EDITABLE:
+    get_requires_for_build_editable = _BACKEND.get_requires_for_build_editable
+    prepare_metadata_for_build_editable = _BACKEND.prepare_metadata_for_build_editable
+    build_editable = _BACKEND.build_editable
 
 
 # The legacy backend

setuptools/command/build.py

@@ -44,20 +44,20 @@ class SubCommand(Protocol):
     1. ``setuptools`` will set the ``editable_mode`` flag will be set to ``True``
     2. ``setuptools`` will execute the ``run()`` command.
 
-         .. important::
-            Subcommands **SHOULD** take advantage of ``editable_mode=True`` to adequate
-            its behaviour or perform optimisations.
-
-            For example, if a subcommand don't need to generate any extra file and
-            everything it does is to copy a source file into the build directory,
-            ``run()`` **SHOULD** simply "early return".
-
-            Similarly, if the subcommand creates files that would be placed alongside
-            Python files in the final distribution, during an editable install
-            the command **SHOULD** generate these files "in place" (i.e. write them to
-            the original source directory, instead of using the build directory).
-            Note that ``get_output_mapping()`` should reflect that and include mappings
-            for "in place" builds accordingly.
+       .. important::
+          Subcommands **SHOULD** take advantage of ``editable_mode=True`` to adequate
+          its behaviour or perform optimisations.
+
+          For example, if a subcommand don't need to generate any extra file and
+          everything it does is to copy a source file into the build directory,
+          ``run()`` **SHOULD** simply "early return".
+
+          Similarly, if the subcommand creates files that would be placed alongside
+          Python files in the final distribution, during an editable install
+          the command **SHOULD** generate these files "in place" (i.e. write them to
+          the original source directory, instead of using the build directory).
+          Note that ``get_output_mapping()`` should reflect that and include mappings
+          for "in place" builds accordingly.
 
     3. ``setuptools`` use any knowledge it can derive from the return values of
        ``get_outputs()`` and ``get_output_mapping()`` to create an editable wheel.