From 8e7ba62557921cb38fafb257a5c2b33c86e750f9 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Tue, 28 Jun 2022 09:10:08 +1000 Subject: [PATCH 01/30] add initial translation of the flow document from the MarkDown original --- source/flow.rst | 209 +++++++++++++++++++++++++++++++++++++++++++++++ source/index.rst | 2 + 2 files changed, 211 insertions(+) create mode 100644 source/flow.rst diff --git a/source/flow.rst b/source/flow.rst new file mode 100644 index 000000000..e59acd987 --- /dev/null +++ b/source/flow.rst @@ -0,0 +1,209 @@ +================== +The Packaging Flow +================== + +The document aims to outline the flow involved in publishing a package, +usually to the `Python Package Index (PyPI)`_ + +.. _Python Package Index (PyPI): https://pypi.org/ + +While the `tutorial`_ +walks through the process of preparing a simple package for release +it does not fully enumerate what steps and files are required, +and for what purpose. + +.. _tutorial: https://packaging.python.org/en/latest/tutorials/installing-packages/ + +This guide is aimed at package publishers, and for simplification +presumes that that is also the package author. +Also, we will use the word package to cover modules as well. + +To publish a package there needs to be a flow from the author's +source code to an end user's Python installation. +The steps to achieve this are as follows: + +- have a source tree containing the package and associated metadata + describing the package (name, version and so forth), typically a checkout + from a version control system (VCS) + +- prepare a configuration file describing the package metadata and how to + create the build artifacts; for many packages this will be a static + ``pyproject.toml`` file in the source tree, + simple and hand maintained as part of the source tree + +- create build artifacts to be sent to the package distribution service + (usually PyPI); this will normally be a `source distribution ("sdist")`_ + and a number of `built distributions ("wheel" files)`_ + often there is just one generic wheel for a pure Python package; + these are made by a build tool/system using the configuration file + from the previous step + +.. _source distribution ("sdist"): https://packaging.python.org/en/latest/glossary/#term-Source-Distribution-or-sdist +.. _built distributions ("wheel" files): https://packaging.python.org/en/latest/glossary/#term-Built-Distribution + +- upload the build artifacts to the package distribution service + +At that point the package is present on the package distribution service. +To use the package, end users must: + +- download one of the package's build artifacts from the package + distribution service + +- install it in their Python installation, usually in its ``site-packages`` + directory; this install step may involve a build/compile step which, + if needed, must be described by the package metadata + +These last 2 steps are typically performed by a tool like `pip`_. + +.. _pip: https://pip.pypa.io/en/stable/ + +The Steps In More Detail +======================== + +The steps above are described in more detail below. + +The Source Tree +--------------- + +The source tree contains the package source code, usually a checkout from a VCS. +The particular version of the code will will typically be from a checkout +based on a tag associated with the version. + +The Configuration File +---------------------- + +The configuration file depends on the tool used to build the build artifacts. +Modern practice is a ``pyproject.toml`` file in `TOML format`_ +whose contents are specified by `PEP 517`_, `PEP 518`_ and `PEP 621`_. + +.. _TOML format: https://github.com/toml-lang/toml +.. _PEP 517: https://peps.python.org/pep-0517/ +.. _PEP 518: https://peps.python.org/pep-0518/ +.. _PEP 621: https://peps.python.org/pep-0621/ + +At a minimum, the ``pyproject.toml`` file needs: +- a ``[project]`` table containing the `Core Metadata`_ for the project + (name, version, author and so forth); + the fields used in ``pyproject.toml`` + are described in `PEP 621`_ +- a ``[build-system]`` table specifying your build tool, + which you will use to create the build artifacts + and which an installer such as `pip` will use + to complete an install from a source distribution + +.. _Core Metadata: https://packaging.python.org/en/latest/specifications/core-metadata/ + +The Build System +---------------- + +The build tool itself is specified by the required table ``[build-system]``. +There are several choices available, including but not limited to: +`flit`_, `hatch`_, `pdm`_, `poetry`_, `setuptools`_, `trampolim`_, +`whey`_. + +.. _flit: https://pypi.org/project/flit/ +.. _hatch: https://github.com/ofek/hatch +.. _pdm: https://pypi.org/project/pdm/ +.. _poetry: https://pypi.org/project/poetry/ +.. _setuptools: https://pypi.org/project/setuptools/ +.. _trampolim: https://pypi.org/project/trampolim/ +.. _whey: https://pypi.org/project/whey/ + +For example, ere is a table for using ``setuptools`` (see the `Setuptools documentation`_):: + + [build-system] + requires = [ + "setuptools >= 61.0", + "trove-classifiers", + "wheel", + ] + build-backend = "setuptools.build_meta" + +.. _Setuptools documentation: https://setuptools.pypa.io/en/latest/userguide/index.html + +or for ``flit`` (see the `Flit documentation`_):: + + [build-system] + requires = ["flit_core >=3.2,<4"] + build-backend = "flit_core.buildapi" + +.. _Flit documentation: https://flit.pypa.io/en/latest/ + +With such a table in the ``pyproject.toml`` file a tool like `build`_ +can run your chosen build system to create the build artifacts +and an install tool like `pip` can fetch and run the build system +when installing a source distribution. + +.. _build: https://pypi.org/project/build/ + +The particular build system you choose dictates what additional information is required. +For example, ``setuptools`` wants a ``setup.cfg`` file containing package metadata +and it is also prudent to provide a stub ``setup.py`` containing:: + + from setuptools import setup + setup() + +or equivalent (``setuptools`` is moving away from actually *running* the ``setup.py`` file directly). + +Build Artifacts: the Source Distribution (sdist) +------------------------------------------------ + +A source distribution contains enough to install the package from source +on an end user's system. +As such it needs the package source +and may well also include tests and documentation. +These are useful for end users wanting to develop your sources +and for end user systems where some local compilation step is required, +for example for a C extension. + +A build system will know how to create one of these, +and the ``build`` package knows how to invoke your build system to create one:: + + python3 -m build --sdist source-tree-directory + +Or, of course, you can invoke your build tool directly. + +Build Artifacts: the Built Distributions (wheels) +------------------------------------------------- + +A built distribution contains the completed files needed for a specific +end user system; no compilations steps are required during the install +and the wheel file can simply be unpacked into the right place. +This makes these faster and more convenient for end users; +tools like ``pip`` will fall back to the source distribtion +if a suitable wheel file is not available. +A pure Python package only needs one wheel for "generic" systems. + +A build system will know how to create one of these, +and the ``build`` package knows how to invoke your build system to create one:: + + python3 -m build --wheel source-tree-directory + +Or, of course, you can invoke your build tool directly. + +The default behaviour of ``build`` is to make both an sdist and a wheel; +the above examples are deliberately specific. + +Upload to the Package Distribution Service +------------------------------------------ + +The `twine tool`_ can upload build artifact files to PyPI for distribution, +for example with a command like:: + + twine upload dist/package-name-version.tar.gz dist/package-name-version-py3-none-any.whl + +.. _twine tool: https://pypi.org/project/twine/ + +Some build tools will also include their own upload facilities. + +Download/Install +---------------- + +Now that the package is published, +end users then download and install the package. +Typically this is done with ``pip``, ideally wiith a command line like:: + + python3 -m pip install package-name + +where ``python3`` is the python executable which is to have +``package-name`` installed. diff --git a/source/index.rst b/source/index.rst index 20e74d918..f2b88fe56 100644 --- a/source/index.rst +++ b/source/index.rst @@ -37,6 +37,8 @@ Get started Essential tools and concepts for working within the Python development ecosystem are covered in our :doc:`tutorials/index` section: +* To get an overview of the publishing flow used to publish your code, see the + :doc:`packaging flow ` * To learn how to install packages, see the :doc:`tutorial on installing packages ` * To learn how to manage dependencies in a version controlled project, see the From 93e0b22e67499488f2f35149518635e069d18683 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Tue, 28 Jun 2022 10:15:00 +1000 Subject: [PATCH 02/30] source/index.rst: add the flow to the toctree --- source/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/source/index.rst b/source/index.rst index f2b88fe56..e2769a29a 100644 --- a/source/index.rst +++ b/source/index.rst @@ -11,6 +11,7 @@ Python Packaging User Guide :hidden: overview + flow tutorials/index guides/index discussions/index From 4d7282c05a92622545236fd5cf3eb4c6432c5d41 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Tue, 28 Jun 2022 10:15:26 +1000 Subject: [PATCH 03/30] source/flow.rst: minor formatting and typo fixes --- source/flow.rst | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index e59acd987..f6aa2ce56 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -38,11 +38,11 @@ The steps to achieve this are as follows: these are made by a build tool/system using the configuration file from the previous step +- upload the build artifacts to the package distribution service + .. _source distribution ("sdist"): https://packaging.python.org/en/latest/glossary/#term-Source-Distribution-or-sdist .. _built distributions ("wheel" files): https://packaging.python.org/en/latest/glossary/#term-Built-Distribution -- upload the build artifacts to the package distribution service - At that point the package is present on the package distribution service. To use the package, end users must: @@ -74,7 +74,9 @@ The Configuration File The configuration file depends on the tool used to build the build artifacts. Modern practice is a ``pyproject.toml`` file in `TOML format`_ -whose contents are specified by `PEP 517`_, `PEP 518`_ and `PEP 621`_. +whose contents are specified by +`PEP 517`_ and `PEP 518`_ (specifying a build system), +and `PEP 621`_ (storing project metadata in ``pyproject.toml``). .. _TOML format: https://github.com/toml-lang/toml .. _PEP 517: https://peps.python.org/pep-0517/ @@ -82,11 +84,13 @@ whose contents are specified by `PEP 517`_, `PEP 518`_ and `PEP 621`_. .. _PEP 621: https://peps.python.org/pep-0621/ At a minimum, the ``pyproject.toml`` file needs: -- a ``[project]`` table containing the `Core Metadata`_ for the project + +* a ``[project]`` table containing the `Core Metadata`_ for the project (name, version, author and so forth); the fields used in ``pyproject.toml`` are described in `PEP 621`_ -- a ``[build-system]`` table specifying your build tool, + +* a ``[build-system]`` table specifying your build tool, which you will use to create the build artifacts and which an installer such as `pip` will use to complete an install from a source distribution @@ -109,7 +113,7 @@ There are several choices available, including but not limited to: .. _trampolim: https://pypi.org/project/trampolim/ .. _whey: https://pypi.org/project/whey/ -For example, ere is a table for using ``setuptools`` (see the `Setuptools documentation`_):: +For example, here is a table for using ``setuptools`` (see the `Setuptools documentation`_):: [build-system] requires = [ From d169cf7ef26bdba1e9f4f4ed804bce40eb44d116 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Tue, 28 Jun 2022 22:22:07 +1000 Subject: [PATCH 04/30] flow.rst: some suggestions from @ abravalheri --- source/flow.rst | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index f6aa2ce56..2ccacc135 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -3,7 +3,7 @@ The Packaging Flow ================== The document aims to outline the flow involved in publishing a package, -usually to the `Python Package Index (PyPI)`_ +usually to the `Python Package Index (PyPI)`_. .. _Python Package Index (PyPI): https://pypi.org/ @@ -116,11 +116,7 @@ There are several choices available, including but not limited to: For example, here is a table for using ``setuptools`` (see the `Setuptools documentation`_):: [build-system] - requires = [ - "setuptools >= 61.0", - "trove-classifiers", - "wheel", - ] + requires = ["setuptools>=61.2""] build-backend = "setuptools.build_meta" .. _Setuptools documentation: https://setuptools.pypa.io/en/latest/userguide/index.html @@ -141,8 +137,9 @@ when installing a source distribution. .. _build: https://pypi.org/project/build/ The particular build system you choose dictates what additional information is required. -For example, ``setuptools`` wants a ``setup.cfg`` file containing package metadata -and it is also prudent to provide a stub ``setup.py`` containing:: +For example, if using ``setuptools`` you might want to add a ``setup.cfg`` file +containing package metadata and/or a small ``setup.py`` stub for backward +compatibility, for example:: from setuptools import setup setup() From ae3c2a33c14f4e0f3b91b344d39ec52c6da9f0a6 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Wed, 29 Jun 2022 08:31:37 +1000 Subject: [PATCH 05/30] flow.rst: small clarification that this is about distribution packages, suggestion from @sinoroc --- source/flow.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 2ccacc135..e81464024 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -2,9 +2,10 @@ The Packaging Flow ================== -The document aims to outline the flow involved in publishing a package, -usually to the `Python Package Index (PyPI)`_. +The document aims to outline the flow involved in publishing/distributing +a `distribution package`_, usually to the `Python Package Index (PyPI)`_. +.. _distribution package: https://packaging.python.org/en/latest/glossary/#term-Distribution-Package .. _Python Package Index (PyPI): https://pypi.org/ While the `tutorial`_ From 1a14289f71f58baa44882d34d337efa1cae46567 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Wed, 29 Jun 2022 08:36:35 +1000 Subject: [PATCH 06/30] undo previous commit - this isn't just about distribution packages --- source/flow.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index e81464024..2ccacc135 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -2,10 +2,9 @@ The Packaging Flow ================== -The document aims to outline the flow involved in publishing/distributing -a `distribution package`_, usually to the `Python Package Index (PyPI)`_. +The document aims to outline the flow involved in publishing a package, +usually to the `Python Package Index (PyPI)`_. -.. _distribution package: https://packaging.python.org/en/latest/glossary/#term-Distribution-Package .. _Python Package Index (PyPI): https://pypi.org/ While the `tutorial`_ From 2322aa62cf2f02a6186bd92bf27a9e6b3ea83262 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Wed, 29 Jun 2022 10:03:29 +1000 Subject: [PATCH 07/30] flow.rst: I was wrong, restoring @sinoroc's suggestion --- source/flow.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 2ccacc135..e81464024 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -2,9 +2,10 @@ The Packaging Flow ================== -The document aims to outline the flow involved in publishing a package, -usually to the `Python Package Index (PyPI)`_. +The document aims to outline the flow involved in publishing/distributing +a `distribution package`_, usually to the `Python Package Index (PyPI)`_. +.. _distribution package: https://packaging.python.org/en/latest/glossary/#term-Distribution-Package .. _Python Package Index (PyPI): https://pypi.org/ While the `tutorial`_ From 7ea231f54f58e7d7efe0cd3b153c90609b0f3c31 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Thu, 30 Jun 2022 11:47:16 +1000 Subject: [PATCH 08/30] flow.rst: The Configuration File: be less prescriptive - suggestion by @abravalheri --- source/flow.rst | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index e81464024..50c625786 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -86,17 +86,22 @@ and `PEP 621`_ (storing project metadata in ``pyproject.toml``). At a minimum, the ``pyproject.toml`` file needs: -* a ``[project]`` table containing the `Core Metadata`_ for the project - (name, version, author and so forth); - the fields used in ``pyproject.toml`` - are described in `PEP 621`_ - * a ``[build-system]`` table specifying your build tool, - which you will use to create the build artifacts - and which an installer such as `pip` will use + which *you* will use to create the build artifacts + and which an *installer* such as `pip` will use to complete an install from a source distribution +Very frequently you might also specify in ``pyproject.toml`` +(depending on the tools you use): + +* a ``[project]`` table containing project `Core Metadata`_ + (name, version, author and so forth); + see `Declaring project metadata`_ for more detail + +* a ``[tool]`` table containing tool-specific configuration options + .. _Core Metadata: https://packaging.python.org/en/latest/specifications/core-metadata/ +.. _Declaring project metadata: https://packaging.python.org/en/latest/specifications/declaring-project-metadata/ The Build System ---------------- From 34a91e976de9b2caf74fe3e1653fd55761cc5f1b Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Thu, 30 Jun 2022 20:57:18 +1000 Subject: [PATCH 09/30] flow.rst: The Build System: small elaboration suggested by @pfmoore --- source/flow.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/source/flow.rst b/source/flow.rst index 50c625786..6f0bdd2b1 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -107,6 +107,11 @@ The Build System ---------------- The build tool itself is specified by the required table ``[build-system]``. +Most build tools provide a "build frontend" such as a command line mode +for creating the build artifacts. +All can also be used via the `build`_ module +which understands the ``[build-system]`` table. + There are several choices available, including but not limited to: `flit`_, `hatch`_, `pdm`_, `poetry`_, `setuptools`_, `trampolim`_, `whey`_. From 1f18e2976d2ca32690b3433f1ba2db309461f285 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Thu, 30 Jun 2022 21:03:41 +1000 Subject: [PATCH 10/30] flow.rst: use :pep: roles for PEP references --- source/flow.rst | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 6f0bdd2b1..706940b32 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -76,13 +76,10 @@ The Configuration File The configuration file depends on the tool used to build the build artifacts. Modern practice is a ``pyproject.toml`` file in `TOML format`_ whose contents are specified by -`PEP 517`_ and `PEP 518`_ (specifying a build system), -and `PEP 621`_ (storing project metadata in ``pyproject.toml``). +:pep:`517`_ and :pep:`518`_ (specifying a build system), +and :pep:`621`_ (storing project metadata in ``pyproject.toml``). .. _TOML format: https://github.com/toml-lang/toml -.. _PEP 517: https://peps.python.org/pep-0517/ -.. _PEP 518: https://peps.python.org/pep-0518/ -.. _PEP 621: https://peps.python.org/pep-0621/ At a minimum, the ``pyproject.toml`` file needs: From d96323fe6902d11739533bac16d6f39fa2872657 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Thu, 30 Jun 2022 21:19:58 +1000 Subject: [PATCH 11/30] flow.rst: convert in tree refs to :doc: --- source/flow.rst | 22 +++++++--------------- 1 file changed, 7 insertions(+), 15 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 706940b32..32c49f81f 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -3,18 +3,16 @@ The Packaging Flow ================== The document aims to outline the flow involved in publishing/distributing -a `distribution package`_, usually to the `Python Package Index (PyPI)`_. +a :term:`distribution package `, +usually to the `Python Package Index (PyPI)`_. -.. _distribution package: https://packaging.python.org/en/latest/glossary/#term-Distribution-Package .. _Python Package Index (PyPI): https://pypi.org/ -While the `tutorial`_ +While the :doc:`tutorial ` walks through the process of preparing a simple package for release it does not fully enumerate what steps and files are required, and for what purpose. -.. _tutorial: https://packaging.python.org/en/latest/tutorials/installing-packages/ - This guide is aimed at package publishers, and for simplification presumes that that is also the package author. Also, we will use the word package to cover modules as well. @@ -33,17 +31,14 @@ The steps to achieve this are as follows: simple and hand maintained as part of the source tree - create build artifacts to be sent to the package distribution service - (usually PyPI); this will normally be a `source distribution ("sdist")`_ - and a number of `built distributions ("wheel" files)`_ + (usually PyPI); this will normally be a :term:`source distribution ("sdist") ` + and a number of :term:`built distributions ("wheel" files) ` often there is just one generic wheel for a pure Python package; these are made by a build tool/system using the configuration file from the previous step - upload the build artifacts to the package distribution service -.. _source distribution ("sdist"): https://packaging.python.org/en/latest/glossary/#term-Source-Distribution-or-sdist -.. _built distributions ("wheel" files): https://packaging.python.org/en/latest/glossary/#term-Built-Distribution - At that point the package is present on the package distribution service. To use the package, end users must: @@ -91,15 +86,12 @@ At a minimum, the ``pyproject.toml`` file needs: Very frequently you might also specify in ``pyproject.toml`` (depending on the tools you use): -* a ``[project]`` table containing project `Core Metadata`_ +* a ``[project]`` table containing project :doc:`Core Metadata ` (name, version, author and so forth); - see `Declaring project metadata`_ for more detail + see :doc:`Declaring project metadata ` for more detail * a ``[tool]`` table containing tool-specific configuration options -.. _Core Metadata: https://packaging.python.org/en/latest/specifications/core-metadata/ -.. _Declaring project metadata: https://packaging.python.org/en/latest/specifications/declaring-project-metadata/ - The Build System ---------------- From a9161aa6cccec85de7e47d7ec96f448a7466ee1d Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Thu, 30 Jun 2022 21:27:47 +1000 Subject: [PATCH 12/30] flow.rst: small fixups --- source/flow.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 32c49f81f..642d93d88 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -31,7 +31,7 @@ The steps to achieve this are as follows: simple and hand maintained as part of the source tree - create build artifacts to be sent to the package distribution service - (usually PyPI); this will normally be a :term:`source distribution ("sdist") ` + (usually PyPI); this will normally be a :term:`source distribution ("sdist") ` and a number of :term:`built distributions ("wheel" files) ` often there is just one generic wheel for a pure Python package; these are made by a build tool/system using the configuration file @@ -71,8 +71,8 @@ The Configuration File The configuration file depends on the tool used to build the build artifacts. Modern practice is a ``pyproject.toml`` file in `TOML format`_ whose contents are specified by -:pep:`517`_ and :pep:`518`_ (specifying a build system), -and :pep:`621`_ (storing project metadata in ``pyproject.toml``). +:pep:`517` and :pep:`518` (specifying a build system), +and :pep:`621` (storing project metadata in ``pyproject.toml``). .. _TOML format: https://github.com/toml-lang/toml From 28d909281ee202546a54ff28d35ce7483f04eb16 Mon Sep 17 00:00:00 2001 From: Brian Rutledge Date: Thu, 21 Jul 2022 07:26:44 -0400 Subject: [PATCH 13/30] Lots of copy edits and TODOs --- source/flow.rst | 248 +++++++++++++++++++++++++---------------------- source/index.rst | 2 +- 2 files changed, 132 insertions(+), 118 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 642d93d88..01adf45f4 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -1,119 +1,116 @@ +.. TODO: Reformat to 78 character line length used in in other docs + ================== The Packaging Flow ================== The document aims to outline the flow involved in publishing/distributing a :term:`distribution package `, -usually to the `Python Package Index (PyPI)`_. +usually to the `Python Package Index (PyPI)`_. It is written for package +publishers, who are assumed to be the package author. .. _Python Package Index (PyPI): https://pypi.org/ -While the :doc:`tutorial ` -walks through the process of preparing a simple package for release +While the :doc:`tutorial ` +walks through the process of preparing a simple package for release, it does not fully enumerate what steps and files are required, and for what purpose. -This guide is aimed at package publishers, and for simplification -presumes that that is also the package author. -Also, we will use the word package to cover modules as well. - -To publish a package there needs to be a flow from the author's -source code to an end user's Python installation. -The steps to achieve this are as follows: +Publishing a package requires a flow from the author's source code to an end +user's Python environment. The steps to achieve this are: -- have a source tree containing the package and associated metadata - describing the package (name, version and so forth), typically a checkout - from a version control system (VCS) +- Have a source tree containing the package. This is typically a checkout from + a version control system (VCS). -- prepare a configuration file describing the package metadata and how to - create the build artifacts; for many packages this will be a static - ``pyproject.toml`` file in the source tree, - simple and hand maintained as part of the source tree +- Prepare a configuration file describing the package metadata + (name, version and so forth) and how to create the build artifacts. + For most packages, this will be a :file:`pyproject.toml` file, + maintained maunally in the source tree. -- create build artifacts to be sent to the package distribution service - (usually PyPI); this will normally be a :term:`source distribution ("sdist") ` - and a number of :term:`built distributions ("wheel" files) ` - often there is just one generic wheel for a pure Python package; - these are made by a build tool/system using the configuration file - from the previous step +- Create build artifacts to be sent to the package distribution service + (usually PyPI); these will normally be a + :term:`source distribution ("sdist") ` + and one or more :term:`built distributions ("wheels") `. + These are made by a build tool using the configuration file + from the previous step. Often there is just one generic wheel for a + pure Python package. -- upload the build artifacts to the package distribution service +- Upload the build artifacts to the package distribution service. -At that point the package is present on the package distribution service. +At that point, the package is present on the package distribution service. To use the package, end users must: -- download one of the package's build artifacts from the package - distribution service - -- install it in their Python installation, usually in its ``site-packages`` - directory; this install step may involve a build/compile step which, - if needed, must be described by the package metadata - -These last 2 steps are typically performed by a tool like `pip`_. - -.. _pip: https://pip.pypa.io/en/stable/ +- Download one of the package's build artifacts from the package + distribution service. -The Steps In More Detail -======================== +- Install it in their Python environment, usually in its ``site-packages`` + directory. This step may involve a build/compile step which, + if needed, must be described by the package metadata. The steps above are described in more detail below. -The Source Tree ---------------- +The source tree +=============== -The source tree contains the package source code, usually a checkout from a VCS. -The particular version of the code will will typically be from a checkout -based on a tag associated with the version. +The source tree contains the package source code, usually a checkout from a +VCS. The particular version of the code used to create the build artifacts +will typically be a checkout based on a tag associated with the version. -The Configuration File ----------------------- +The configuration file +====================== -The configuration file depends on the tool used to build the build artifacts. -Modern practice is a ``pyproject.toml`` file in `TOML format`_ -whose contents are specified by -:pep:`517` and :pep:`518` (specifying a build system), -and :pep:`621` (storing project metadata in ``pyproject.toml``). +.. TODO: Are the PEP links useful? + +The configuration file depends on the tool used to create the build artifacts. +The standard practice is to use a :file:`pyproject.toml` file in the `TOML +format`_ whose contents are specified by :pep:`517`, :pep:`518`, and +:pep:`621`. .. _TOML format: https://github.com/toml-lang/toml -At a minimum, the ``pyproject.toml`` file needs: +.. TODO: Normalize "you" (a person) and "installer" (a tool) + +At a minimum, the :file:`pyproject.toml` file needs a ``[build-system]`` table +specifying your build tool, which *you* will use to create the build artifacts +and which an *installer* such as ``pip`` will use to install from +a source distribution. -* a ``[build-system]`` table specifying your build tool, - which *you* will use to create the build artifacts - and which an *installer* such as `pip` will use - to complete an install from a source distribution +.. TODO: Move "build tool" content here? -Very frequently you might also specify in ``pyproject.toml`` -(depending on the tools you use): +Depending on the tools you use, you might also specify: -* a ``[project]`` table containing project :doc:`Core Metadata ` - (name, version, author and so forth); - see :doc:`Declaring project metadata ` for more detail +* a ``[project]`` table containing project + :doc:`Core Metadata ` + (name, version, author and so forth); see + :doc:`Declaring project metadata ` + for more detail * a ``[tool]`` table containing tool-specific configuration options -The Build System ----------------- +The build tool +-------------- -The build tool itself is specified by the required table ``[build-system]``. -Most build tools provide a "build frontend" such as a command line mode -for creating the build artifacts. -All can also be used via the `build`_ module -which understands the ``[build-system]`` table. +.. TODO: Instead of "build tool", should we use "build backend"? +.. TODO: Link tools to project summaries (key_projects.rst), e.g. :ref:`flit`? + +The build tool itself is specified by the required table ``[build-system]``. There are several choices available, including but not limited to: `flit`_, `hatch`_, `pdm`_, `poetry`_, `setuptools`_, `trampolim`_, `whey`_. .. _flit: https://pypi.org/project/flit/ -.. _hatch: https://github.com/ofek/hatch +.. _hatch: https://github.com/pypa/hatch .. _pdm: https://pypi.org/project/pdm/ .. _poetry: https://pypi.org/project/poetry/ .. _setuptools: https://pypi.org/project/setuptools/ .. _trampolim: https://pypi.org/project/trampolim/ .. _whey: https://pypi.org/project/whey/ -For example, here is a table for using ``setuptools`` (see the `Setuptools documentation`_):: +.. TODO: Use hatchling and setuptools, ala the packaging tutorial? + +For example, here is a table for using ``setuptools`` (see the `Setuptools +documentation`_):: [build-system] requires = ["setuptools>=61.2""] @@ -121,7 +118,7 @@ For example, here is a table for using ``setuptools`` (see the `Setuptools docum .. _Setuptools documentation: https://setuptools.pypa.io/en/latest/userguide/index.html -or for ``flit`` (see the `Flit documentation`_):: +or for using ``flit`` (see the `Flit documentation`_):: [build-system] requires = ["flit_core >=3.2,<4"] @@ -129,82 +126,99 @@ or for ``flit`` (see the `Flit documentation`_):: .. _Flit documentation: https://flit.pypa.io/en/latest/ -With such a table in the ``pyproject.toml`` file a tool like `build`_ -can run your chosen build system to create the build artifacts -and an install tool like `pip` can fetch and run the build system +.. TODO: Should we use "build frontend" here? + +With such a table in the :file:`pyproject.toml` file, a tool like `build`_ +can run your chosen build tool to create the build artifacts, +and an install tool like ``pip`` can run the build tool when installing a source distribution. .. _build: https://pypi.org/project/build/ -The particular build system you choose dictates what additional information is required. -For example, if using ``setuptools`` you might want to add a ``setup.cfg`` file -containing package metadata and/or a small ``setup.py`` stub for backward -compatibility, for example:: +The particular build tool you choose dictates what additional information is +required in the :file:`pyproject.toml` file. - from setuptools import setup - setup() +Build artifacts +=============== -or equivalent (``setuptools`` is moving away from actually *running* the ``setup.py`` file directly). +The source distribution (sdist) +------------------------------- -Build Artifacts: the Source Distribution (sdist) ------------------------------------------------- +A source distribution contains enough to install the package from source on an +end user's Python environment. As such, it needs the package source, and may +also include tests and documentation. These are useful for end users wanting +to develop your sources, and for end user systems where some local compilation +step is required (such as a C extension). -A source distribution contains enough to install the package from source -on an end user's system. -As such it needs the package source -and may well also include tests and documentation. -These are useful for end users wanting to develop your sources -and for end user systems where some local compilation step is required, -for example for a C extension. +The ``build`` package knows how to invoke your build tool to create one of +these: -A build system will know how to create one of these, -and the ``build`` package knows how to invoke your build system to create one:: +.. code-block:: bash python3 -m build --sdist source-tree-directory -Or, of course, you can invoke your build tool directly. +Or, your build tool may provide its own interface for creating an sdist. + -Build Artifacts: the Built Distributions (wheels) -------------------------------------------------- +The built distributions (wheels) +-------------------------------- -A built distribution contains the completed files needed for a specific -end user system; no compilations steps are required during the install -and the wheel file can simply be unpacked into the right place. -This makes these faster and more convenient for end users; -tools like ``pip`` will fall back to the source distribtion -if a suitable wheel file is not available. -A pure Python package only needs one wheel for "generic" systems. +.. TODO: Clarify "end user system", maybe with an example, e.g. OS and architecture -A build system will know how to create one of these, -and the ``build`` package knows how to invoke your build system to create one:: +A built distribution contains only the files needed for a specific end user +system; no compilation steps are required during the install, and the wheel +file can simply be unpacked into the ``site-packages`` directory. This makes +the install faster and more convenient for end users. A pure Python package +only needs one wheel for "generic" systems. If a suitable wheel file is not +available, tools like ``pip`` will fall back to installing the source +distribtion. + +The ``build`` package knows how to invoke your build tool to create one of +these: + +.. code-block:: bash python3 -m build --wheel source-tree-directory -Or, of course, you can invoke your build tool directly. +Or, your build tool may provide its own interface for creating a wheel. + +.. note:: -The default behaviour of ``build`` is to make both an sdist and a wheel; -the above examples are deliberately specific. + The default behaviour of ``build`` is to make both an sdist and a wheel from + the source in the current directory; the above examples are deliberately + specific. -Upload to the Package Distribution Service ------------------------------------------- +Upload to the package distribution service +========================================== -The `twine tool`_ can upload build artifact files to PyPI for distribution, -for example with a command like:: +The `twine`_ tool can upload build artifacts to PyPI for distribution, using a +command like: + +.. code-block:: bash twine upload dist/package-name-version.tar.gz dist/package-name-version-py3-none-any.whl -.. _twine tool: https://pypi.org/project/twine/ +.. _twine: https://pypi.org/project/twine/ -Some build tools will also include their own upload facilities. +Or, your build tool may provide its own interface for uploading. -Download/Install ----------------- +Download and install +==================== Now that the package is published, -end users then download and install the package. -Typically this is done with ``pip``, ideally wiith a command line like:: +end users can download and install the package into their Python environment. +Typically this is done with `pip`_, using a command like: + +.. code-block:: bash python3 -m pip install package-name -where ``python3`` is the python executable which is to have -``package-name`` installed. +.. _pip: https://pip.pypa.io/en/stable/ + +.. TODO: Briefly describe typical behavior of using a virtual environment, +.. and maybe link to one or more of: +.. tutorials/installing-packages +.. guides/installing-using-pip-and-virtual-environments +.. guides/installing-stand-alone-command-line-tools.html + +.. TODO: Mention poetry, pdm, and pipenv as other methods? \ No newline at end of file diff --git a/source/index.rst b/source/index.rst index e2769a29a..4cd8c8425 100644 --- a/source/index.rst +++ b/source/index.rst @@ -38,7 +38,7 @@ Get started Essential tools and concepts for working within the Python development ecosystem are covered in our :doc:`tutorials/index` section: -* To get an overview of the publishing flow used to publish your code, see the +* To get an overview of the flow used to publish your code, see the :doc:`packaging flow ` * To learn how to install packages, see the :doc:`tutorial on installing packages ` From bcdf5dfbcfe04812ce68eabb3c84ab24196e1538 Mon Sep 17 00:00:00 2001 From: Brian Rutledge Date: Thu, 21 Jul 2022 11:04:38 -0400 Subject: [PATCH 14/30] Set 78 character line length --- source/flow.rst | 61 +++++++++++++++++++++++-------------------------- 1 file changed, 28 insertions(+), 33 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 01adf45f4..fa9a9af00 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -1,20 +1,17 @@ -.. TODO: Reformat to 78 character line length used in in other docs - ================== The Packaging Flow ================== -The document aims to outline the flow involved in publishing/distributing -a :term:`distribution package `, -usually to the `Python Package Index (PyPI)`_. It is written for package -publishers, who are assumed to be the package author. +The document aims to outline the flow involved in publishing/distributing a +:term:`distribution package `, usually to the `Python +Package Index (PyPI)`_. It is written for package publishers, who are assumed +to be the package author. .. _Python Package Index (PyPI): https://pypi.org/ -While the :doc:`tutorial ` -walks through the process of preparing a simple package for release, -it does not fully enumerate what steps and files are required, -and for what purpose. +While the :doc:`tutorial ` walks through the +process of preparing a simple package for release, it does not fully enumerate +what steps and files are required, and for what purpose. Publishing a package requires a flow from the author's source code to an end user's Python environment. The steps to achieve this are: @@ -22,30 +19,30 @@ user's Python environment. The steps to achieve this are: - Have a source tree containing the package. This is typically a checkout from a version control system (VCS). -- Prepare a configuration file describing the package metadata - (name, version and so forth) and how to create the build artifacts. - For most packages, this will be a :file:`pyproject.toml` file, - maintained maunally in the source tree. +- Prepare a configuration file describing the package metadata (name, version + and so forth) and how to create the build artifacts. For most packages, this + will be a :file:`pyproject.toml` file, maintained maunally in the source + tree. - Create build artifacts to be sent to the package distribution service (usually PyPI); these will normally be a :term:`source distribution ("sdist") ` and one or more :term:`built distributions ("wheels") `. - These are made by a build tool using the configuration file - from the previous step. Often there is just one generic wheel for a - pure Python package. + These are made by a build tool using the configuration file from the + previous step. Often there is just one generic wheel for a pure Python + package. - Upload the build artifacts to the package distribution service. At that point, the package is present on the package distribution service. To use the package, end users must: -- Download one of the package's build artifacts from the package - distribution service. +- Download one of the package's build artifacts from the package distribution + service. - Install it in their Python environment, usually in its ``site-packages`` - directory. This step may involve a build/compile step which, - if needed, must be described by the package metadata. + directory. This step may involve a build/compile step which, if needed, must + be described by the package metadata. The steps above are described in more detail below. @@ -72,8 +69,8 @@ format`_ whose contents are specified by :pep:`517`, :pep:`518`, and At a minimum, the :file:`pyproject.toml` file needs a ``[build-system]`` table specifying your build tool, which *you* will use to create the build artifacts -and which an *installer* such as ``pip`` will use to install from -a source distribution. +and which an *installer* such as ``pip`` will use to install from a source +distribution. .. TODO: Move "build tool" content here? @@ -95,9 +92,8 @@ The build tool .. TODO: Link tools to project summaries (key_projects.rst), e.g. :ref:`flit`? The build tool itself is specified by the required table ``[build-system]``. -There are several choices available, including but not limited to: -`flit`_, `hatch`_, `pdm`_, `poetry`_, `setuptools`_, `trampolim`_, -`whey`_. +There are several choices available, including but not limited to: `flit`_, +`hatch`_, `pdm`_, `poetry`_, `setuptools`_, `trampolim`_, `whey`_. .. _flit: https://pypi.org/project/flit/ .. _hatch: https://github.com/pypa/hatch @@ -128,10 +124,9 @@ or for using ``flit`` (see the `Flit documentation`_):: .. TODO: Should we use "build frontend" here? -With such a table in the :file:`pyproject.toml` file, a tool like `build`_ -can run your chosen build tool to create the build artifacts, -and an install tool like ``pip`` can run the build tool -when installing a source distribution. +With such a table in the :file:`pyproject.toml` file, a tool like `build`_ can +run your chosen build tool to create the build artifacts, and an install tool +like ``pip`` can run the build tool when installing a source distribution. .. _build: https://pypi.org/project/build/ @@ -205,9 +200,9 @@ Or, your build tool may provide its own interface for uploading. Download and install ==================== -Now that the package is published, -end users can download and install the package into their Python environment. -Typically this is done with `pip`_, using a command like: +Now that the package is published, end users can download and install the +package into their Python environment. Typically this is done with `pip`_, +using a command like: .. code-block:: bash From df3eb1637f0466f23b2b089c1eacd6169f8e253c Mon Sep 17 00:00:00 2001 From: Brian Rutledge Date: Thu, 21 Jul 2022 11:02:29 -0400 Subject: [PATCH 15/30] Condense build-system description --- source/flow.rst | 42 +++++++++++++++--------------------------- 1 file changed, 15 insertions(+), 27 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index fa9a9af00..83b6f71ee 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -65,35 +65,15 @@ format`_ whose contents are specified by :pep:`517`, :pep:`518`, and .. _TOML format: https://github.com/toml-lang/toml -.. TODO: Normalize "you" (a person) and "installer" (a tool) - -At a minimum, the :file:`pyproject.toml` file needs a ``[build-system]`` table -specifying your build tool, which *you* will use to create the build artifacts -and which an *installer* such as ``pip`` will use to install from a source -distribution. - -.. TODO: Move "build tool" content here? - -Depending on the tools you use, you might also specify: - -* a ``[project]`` table containing project - :doc:`Core Metadata ` - (name, version, author and so forth); see - :doc:`Declaring project metadata ` - for more detail - -* a ``[tool]`` table containing tool-specific configuration options - -The build tool --------------- - .. TODO: Instead of "build tool", should we use "build backend"? - .. TODO: Link tools to project summaries (key_projects.rst), e.g. :ref:`flit`? -The build tool itself is specified by the required table ``[build-system]``. -There are several choices available, including but not limited to: `flit`_, -`hatch`_, `pdm`_, `poetry`_, `setuptools`_, `trampolim`_, `whey`_. +At a minimum, the :file:`pyproject.toml` file needs a ``[build-system]`` table +specifying your build tool, which you will use to create the build artifacts, +and which an install tool such as ``pip`` will use to install your package +from a source distribution. There are many build tools available, including +but not limited to: `flit`_, `hatch`_, `pdm`_, `poetry`_, `setuptools`_, +`trampolim`_, `whey`_. .. _flit: https://pypi.org/project/flit/ .. _hatch: https://github.com/pypa/hatch @@ -131,7 +111,15 @@ like ``pip`` can run the build tool when installing a source distribution. .. _build: https://pypi.org/project/build/ The particular build tool you choose dictates what additional information is -required in the :file:`pyproject.toml` file. +required in the :file:`pyproject.toml` file. For example, you might specify: + +* a ``[project]`` table containing project + :doc:`Core Metadata ` + (name, version, author and so forth); see + :doc:`Declaring project metadata ` + for more detail + +* a ``[tool]`` table containing tool-specific configuration options Build artifacts =============== From 962f464ca147186594706f623c19780bf53174c7 Mon Sep 17 00:00:00 2001 From: Brian Rutledge Date: Thu, 21 Jul 2022 11:11:52 -0400 Subject: [PATCH 16/30] Tweak project links --- source/flow.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 83b6f71ee..87b2d700c 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -75,8 +75,9 @@ from a source distribution. There are many build tools available, including but not limited to: `flit`_, `hatch`_, `pdm`_, `poetry`_, `setuptools`_, `trampolim`_, `whey`_. +.. _pip: https://pip.pypa.io/en/stable/ .. _flit: https://pypi.org/project/flit/ -.. _hatch: https://github.com/pypa/hatch +.. _hatch: https://pypi.org/project/hatch/ .. _pdm: https://pypi.org/project/pdm/ .. _poetry: https://pypi.org/project/poetry/ .. _setuptools: https://pypi.org/project/setuptools/ @@ -189,15 +190,13 @@ Download and install ==================== Now that the package is published, end users can download and install the -package into their Python environment. Typically this is done with `pip`_, +package into their Python environment. Typically this is done with ``pip``, using a command like: .. code-block:: bash python3 -m pip install package-name -.. _pip: https://pip.pypa.io/en/stable/ - .. TODO: Briefly describe typical behavior of using a virtual environment, .. and maybe link to one or more of: .. tutorials/installing-packages From 9b8c413d59f65d3a4210baae5665854b74668a00 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Tue, 26 Jul 2022 20:55:56 +1000 Subject: [PATCH 17/30] flow.rst: restore mention of "pip" in the end user steps --- source/flow.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/source/flow.rst b/source/flow.rst index 87b2d700c..84864015a 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -44,6 +44,8 @@ To use the package, end users must: directory. This step may involve a build/compile step which, if needed, must be described by the package metadata. +These last 2 steps are typically performed by a tool like :ref:`pip`. + The steps above are described in more detail below. The source tree From f14c2cdf510103b13f53092ae52cc93c1d9caf44 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Tue, 26 Jul 2022 20:57:25 +1000 Subject: [PATCH 18/30] flow.rst: no, the PEP links are indeed not useful; I've mentioned a PEP elsewhere in context though --- source/flow.rst | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 84864015a..57172baaa 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -58,12 +58,9 @@ will typically be a checkout based on a tag associated with the version. The configuration file ====================== -.. TODO: Are the PEP links useful? - The configuration file depends on the tool used to create the build artifacts. The standard practice is to use a :file:`pyproject.toml` file in the `TOML -format`_ whose contents are specified by :pep:`517`, :pep:`518`, and -:pep:`621`. +format`_. .. _TOML format: https://github.com/toml-lang/toml From 17cf3e542a35db8e1b9ce213039ea26830067564 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Tue, 26 Jul 2022 20:58:38 +1000 Subject: [PATCH 19/30] flow.rst: replace tool links with references to the key projects file --- source/flow.rst | 20 +++++++------------- 1 file changed, 7 insertions(+), 13 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 57172baaa..b0d607220 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -65,21 +65,15 @@ format`_. .. _TOML format: https://github.com/toml-lang/toml .. TODO: Instead of "build tool", should we use "build backend"? -.. TODO: Link tools to project summaries (key_projects.rst), e.g. :ref:`flit`? - At a minimum, the :file:`pyproject.toml` file needs a ``[build-system]`` table specifying your build tool, which you will use to create the build artifacts, -and which an install tool such as ``pip`` will use to install your package -from a source distribution. There are many build tools available, including -but not limited to: `flit`_, `hatch`_, `pdm`_, `poetry`_, `setuptools`_, -`trampolim`_, `whey`_. - -.. _pip: https://pip.pypa.io/en/stable/ -.. _flit: https://pypi.org/project/flit/ -.. _hatch: https://pypi.org/project/hatch/ -.. _pdm: https://pypi.org/project/pdm/ -.. _poetry: https://pypi.org/project/poetry/ -.. _setuptools: https://pypi.org/project/setuptools/ +and which an install tool such as :ref:`pip` will use to install your package +from a source distribution. + +There are many build tools available, including +but not limited to: :ref:`flit`, :ref:`hatch`, :ref:`pdm`, +:ref:`poetry`, :ref:`setuptools`, `trampolim`_, `whey`_. + .. _trampolim: https://pypi.org/project/trampolim/ .. _whey: https://pypi.org/project/whey/ From 9ff8f37f001593b238fed2453f89c702ed856ad0 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Tue, 26 Jul 2022 21:00:13 +1000 Subject: [PATCH 20/30] flow.rst: mention build frontends and backends --- source/flow.rst | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index b0d607220..8cffce3dd 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -64,7 +64,6 @@ format`_. .. _TOML format: https://github.com/toml-lang/toml -.. TODO: Instead of "build tool", should we use "build backend"? At a minimum, the :file:`pyproject.toml` file needs a ``[build-system]`` table specifying your build tool, which you will use to create the build artifacts, and which an install tool such as :ref:`pip` will use to install your package @@ -78,6 +77,12 @@ but not limited to: :ref:`flit`, :ref:`hatch`, :ref:`pdm`, .. _whey: https://pypi.org/project/whey/ .. TODO: Use hatchling and setuptools, ala the packaging tutorial? +Usually a build tool provides both +a "build frontend" (something you can invoke, such as a command) +and a "build backend" (something a program can call, as specified by :pep:`517`). + +For example, both :ref:`pip` and :ref:`build` know how to invoke your build tool +from the information in ``[build-system]`` table. For example, here is a table for using ``setuptools`` (see the `Setuptools documentation`_):: @@ -99,8 +104,8 @@ or for using ``flit`` (see the `Flit documentation`_):: .. TODO: Should we use "build frontend" here? With such a table in the :file:`pyproject.toml` file, a tool like `build`_ can -run your chosen build tool to create the build artifacts, and an install tool -like ``pip`` can run the build tool when installing a source distribution. +run your chosen build tool's backend to create the build artifacts, and an install tool +like ``pip`` can run the build backend when installing a source distribution. .. _build: https://pypi.org/project/build/ From d9e1c27b08fa02e9b7a55095240fab6eaa9e0501 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Tue, 26 Jul 2022 21:01:10 +1000 Subject: [PATCH 21/30] flow.rst: mention all 3 of setuptools, flit and hatchling --- source/flow.rst | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 8cffce3dd..dd34f5722 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -76,7 +76,6 @@ but not limited to: :ref:`flit`, :ref:`hatch`, :ref:`pdm`, .. _trampolim: https://pypi.org/project/trampolim/ .. _whey: https://pypi.org/project/whey/ -.. TODO: Use hatchling and setuptools, ala the packaging tutorial? Usually a build tool provides both a "build frontend" (something you can invoke, such as a command) and a "build backend" (something a program can call, as specified by :pep:`517`). @@ -87,21 +86,33 @@ from the information in ``[build-system]`` table. For example, here is a table for using ``setuptools`` (see the `Setuptools documentation`_):: - [build-system] - requires = ["setuptools>=61.2""] - build-backend = "setuptools.build_meta" + .. code-block:: toml + + [build-system] + requires = ["setuptools>=61.2""] + build-backend = "setuptools.build_meta" .. _Setuptools documentation: https://setuptools.pypa.io/en/latest/userguide/index.html or for using ``flit`` (see the `Flit documentation`_):: - [build-system] - requires = ["flit_core >=3.2,<4"] - build-backend = "flit_core.buildapi" + .. code-block:: toml + + [build-system] + requires = ["flit_core >=3.2,<4"] + build-backend = "flit_core.buildapi" .. _Flit documentation: https://flit.pypa.io/en/latest/ -.. TODO: Should we use "build frontend" here? +or for using :ref:`Hatchling ` (see the `Hatch documentation`_):: + + .. code-block:: toml + + [build-system] + requires = ["hatchling"] + build-backend = "hatchling.build" + +.. _Hatch documentation: https://hatch.pypa.io/latest/ With such a table in the :file:`pyproject.toml` file, a tool like `build`_ can run your chosen build tool's backend to create the build artifacts, and an install tool From fe0b05348dc1509e3a8801a2d99bb760d725ff44 Mon Sep 17 00:00:00 2001 From: Cameron Simpson Date: Tue, 26 Jul 2022 21:02:15 +1000 Subject: [PATCH 22/30] flow.rst: add a paragraph about end user systems how wheel tags can denote them --- source/flow.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index dd34f5722..4be5043d8 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -156,8 +156,6 @@ Or, your build tool may provide its own interface for creating an sdist. The built distributions (wheels) -------------------------------- -.. TODO: Clarify "end user system", maybe with an example, e.g. OS and architecture - A built distribution contains only the files needed for a specific end user system; no compilation steps are required during the install, and the wheel file can simply be unpacked into the ``site-packages`` directory. This makes @@ -166,6 +164,12 @@ only needs one wheel for "generic" systems. If a suitable wheel file is not available, tools like ``pip`` will fall back to installing the source distribtion. +An "end user system" is the end user's local environment. +A pure Python package only needs "generic", but packages with +compiled extensions might build wheels for various environments they support +described by a "python tag" such as `py3`, an "ABI tag" such as `abi3` +and a "platform tag" such as `win32` as detailed by :pep:`425`. + The ``build`` package knows how to invoke your build tool to create one of these: From ac8894f03b8f4a9ccae4d3d4775fd8616ce4c2e1 Mon Sep 17 00:00:00 2001 From: Brian Rutledge Date: Wed, 27 Jul 2022 07:11:14 -0400 Subject: [PATCH 23/30] Reformat and add TODOs --- source/flow.rst | 70 ++++++++++++++++++++++++++++--------------------- 1 file changed, 40 insertions(+), 30 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 4be5043d8..d30952e20 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -44,6 +44,8 @@ To use the package, end users must: directory. This step may involve a build/compile step which, if needed, must be described by the package metadata. +.. TODO: All other tools in the steps above are mentioned later + These last 2 steps are typically performed by a tool like :ref:`pip`. The steps above are described in more detail below. @@ -69,54 +71,60 @@ specifying your build tool, which you will use to create the build artifacts, and which an install tool such as :ref:`pip` will use to install your package from a source distribution. -There are many build tools available, including -but not limited to: :ref:`flit`, :ref:`hatch`, :ref:`pdm`, -:ref:`poetry`, :ref:`setuptools`, `trampolim`_, `whey`_. +.. TODO: Remove trampolim and whey? + +There are many build tools available, including but not limited to: +:ref:`flit`, :ref:`hatch`, :ref:`pdm`, :ref:`poetry`, :ref:`setuptools`, +`trampolim`_, and `whey`_. .. _trampolim: https://pypi.org/project/trampolim/ .. _whey: https://pypi.org/project/whey/ -Usually a build tool provides both -a "build frontend" (something you can invoke, such as a command) -and a "build backend" (something a program can call, as specified by :pep:`517`). +.. TODO: Rework/move this paragraph + +Usually a build tool provides both a "build frontend" (something you can +invoke, such as a command) and a "build backend" (something a program can +call, as specified by :pep:`517`). -For example, both :ref:`pip` and :ref:`build` know how to invoke your build tool -from the information in ``[build-system]`` table. +.. TODO: Only show Hatchling, or use tabs ala packaging tutorial? For example, here is a table for using ``setuptools`` (see the `Setuptools -documentation`_):: +documentation`_): - .. code-block:: toml +.. code-block:: toml - [build-system] - requires = ["setuptools>=61.2""] - build-backend = "setuptools.build_meta" + [build-system] + requires = ["setuptools>=61.2"] + build-backend = "setuptools.build_meta" .. _Setuptools documentation: https://setuptools.pypa.io/en/latest/userguide/index.html -or for using ``flit`` (see the `Flit documentation`_):: +or for using ``flit`` (see the `Flit documentation`_): - .. code-block:: toml +.. code-block:: toml - [build-system] - requires = ["flit_core >=3.2,<4"] - build-backend = "flit_core.buildapi" + [build-system] + requires = ["flit_core >=3.2,<4"] + build-backend = "flit_core.buildapi" .. _Flit documentation: https://flit.pypa.io/en/latest/ -or for using :ref:`Hatchling ` (see the `Hatch documentation`_):: +or for using :ref:`Hatchling ` (see the `Hatch documentation`_): - .. code-block:: toml +.. code-block:: toml - [build-system] - requires = ["hatchling"] - build-backend = "hatchling.build" + [build-system] + requires = ["hatchling"] + build-backend = "hatchling.build" .. _Hatch documentation: https://hatch.pypa.io/latest/ +.. TODO: Use :ref: everywhere + With such a table in the :file:`pyproject.toml` file, a tool like `build`_ can -run your chosen build tool's backend to create the build artifacts, and an install tool -like ``pip`` can run the build backend when installing a source distribution. +run your chosen build tool's backend to create the build artifacts, and an +install tool like ``pip`` can run the build backend when installing a source +distribution. .. _build: https://pypi.org/project/build/ @@ -164,11 +172,13 @@ only needs one wheel for "generic" systems. If a suitable wheel file is not available, tools like ``pip`` will fall back to installing the source distribtion. -An "end user system" is the end user's local environment. -A pure Python package only needs "generic", but packages with -compiled extensions might build wheels for various environments they support -described by a "python tag" such as `py3`, an "ABI tag" such as `abi3` -and a "platform tag" such as `win32` as detailed by :pep:`425`. +.. TODO: Try again; this is too long and specific + +An "end user system" is the end user's local environment. A pure Python +package only needs "generic", but packages with compiled extensions might +build wheels for various environments they support described by a "python tag" +such as ``py3``, an "ABI tag" such as ``abi3`` and a "platform tag" such as +``win32`` as detailed by :pep:`425`. The ``build`` package knows how to invoke your build tool to create one of these: From f72c3d7248aefa31a0355dabd52a9d99682576a6 Mon Sep 17 00:00:00 2001 From: Brian Rutledge Date: Thu, 28 Jul 2022 05:39:14 -0400 Subject: [PATCH 24/30] Link more key projects --- source/flow.rst | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index d30952e20..842f571c6 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -119,15 +119,11 @@ or for using :ref:`Hatchling ` (see the `Hatch documentation`_): .. _Hatch documentation: https://hatch.pypa.io/latest/ -.. TODO: Use :ref: everywhere - -With such a table in the :file:`pyproject.toml` file, a tool like `build`_ can -run your chosen build tool's backend to create the build artifacts, and an +With such a table in the :file:`pyproject.toml` file, a tool like :ref:`build` +can run your chosen build tool's backend to create the build artifacts, and an install tool like ``pip`` can run the build backend when installing a source distribution. -.. _build: https://pypi.org/project/build/ - The particular build tool you choose dictates what additional information is required in the :file:`pyproject.toml` file. For example, you might specify: @@ -198,15 +194,13 @@ Or, your build tool may provide its own interface for creating a wheel. Upload to the package distribution service ========================================== -The `twine`_ tool can upload build artifacts to PyPI for distribution, using a -command like: +The :ref:`twine` tool can upload build artifacts to PyPI for distribution, +using a command like: .. code-block:: bash twine upload dist/package-name-version.tar.gz dist/package-name-version-py3-none-any.whl -.. _twine: https://pypi.org/project/twine/ - Or, your build tool may provide its own interface for uploading. Download and install From 72a5a4f0747f33e24825553eee01645a3fb90316 Mon Sep 17 00:00:00 2001 From: Brian Rutledge Date: Thu, 28 Jul 2022 06:08:07 -0400 Subject: [PATCH 25/30] Rework wheel section --- source/flow.rst | 27 +++++++++++---------------- 1 file changed, 11 insertions(+), 16 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 842f571c6..968beacf8 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -141,7 +141,7 @@ Build artifacts The source distribution (sdist) ------------------------------- -A source distribution contains enough to install the package from source on an +A source distribution contains enough to install the package from source in an end user's Python environment. As such, it needs the package source, and may also include tests and documentation. These are useful for end users wanting to develop your sources, and for end user systems where some local compilation @@ -160,21 +160,16 @@ Or, your build tool may provide its own interface for creating an sdist. The built distributions (wheels) -------------------------------- -A built distribution contains only the files needed for a specific end user -system; no compilation steps are required during the install, and the wheel -file can simply be unpacked into the ``site-packages`` directory. This makes -the install faster and more convenient for end users. A pure Python package -only needs one wheel for "generic" systems. If a suitable wheel file is not -available, tools like ``pip`` will fall back to installing the source -distribtion. - -.. TODO: Try again; this is too long and specific - -An "end user system" is the end user's local environment. A pure Python -package only needs "generic", but packages with compiled extensions might -build wheels for various environments they support described by a "python tag" -such as ``py3``, an "ABI tag" such as ``abi3`` and a "platform tag" such as -``win32`` as detailed by :pep:`425`. +A built distribution contains only the files needed for an end user's Python +environment. No compilation steps are required during the install, and the +wheel file can simply be unpacked into the ``site-packages`` directory. This +makes the install faster and more convenient for end users. + +A pure Python package typically needs only one "generic" wheel. A package with +compiled binary extensions needs a wheel for each supported combination of +Python interprerter, operating system, and CPU architecture that it supports. +If a suitable wheel file is not available, tools like ``pip`` will fall back +to installing the source distribtion. The ``build`` package knows how to invoke your build tool to create one of these: From cb23f4e40299e865a17f64f31cbdc265a22909a9 Mon Sep 17 00:00:00 2001 From: Brian Rutledge Date: Thu, 28 Jul 2022 06:30:35 -0400 Subject: [PATCH 26/30] Rework frontend and backend --- source/flow.rst | 27 ++++++++------------------- 1 file changed, 8 insertions(+), 19 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 968beacf8..0c56458a8 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -67,25 +67,13 @@ format`_. .. _TOML format: https://github.com/toml-lang/toml At a minimum, the :file:`pyproject.toml` file needs a ``[build-system]`` table -specifying your build tool, which you will use to create the build artifacts, -and which an install tool such as :ref:`pip` will use to install your package -from a source distribution. - -.. TODO: Remove trampolim and whey? - -There are many build tools available, including but not limited to: -:ref:`flit`, :ref:`hatch`, :ref:`pdm`, :ref:`poetry`, :ref:`setuptools`, -`trampolim`_, and `whey`_. +specifying your build tool. There are many build tools available, including +but not limited to :ref:`flit`, :ref:`hatch`, :ref:`pdm`, :ref:`poetry`, +:ref:`setuptools`, `trampolim`_, and `whey`_. .. _trampolim: https://pypi.org/project/trampolim/ .. _whey: https://pypi.org/project/whey/ -.. TODO: Rework/move this paragraph - -Usually a build tool provides both a "build frontend" (something you can -invoke, such as a command) and a "build backend" (something a program can -call, as specified by :pep:`517`). - .. TODO: Only show Hatchling, or use tabs ala packaging tutorial? For example, here is a table for using ``setuptools`` (see the `Setuptools @@ -119,10 +107,11 @@ or for using :ref:`Hatchling ` (see the `Hatch documentation`_): .. _Hatch documentation: https://hatch.pypa.io/latest/ -With such a table in the :file:`pyproject.toml` file, a tool like :ref:`build` -can run your chosen build tool's backend to create the build artifacts, and an -install tool like ``pip`` can run the build backend when installing a source -distribution. +With such a table in the :file:`pyproject.toml` file, a "frontend" tool like +:ref:`build` can run your chosen build tool's "backend" to create the build +artifacts. Your build tool may also provide its own frontend. An install tool +like ``pip`` also acts as a frontend when it runs your build tool's backend to +install from a source distribution. The particular build tool you choose dictates what additional information is required in the :file:`pyproject.toml` file. For example, you might specify: From d8b49afddcee55d90e58e47842ecc362b92aed9d Mon Sep 17 00:00:00 2001 From: Brian Rutledge Date: Thu, 28 Jul 2022 08:03:49 -0400 Subject: [PATCH 27/30] Simplify example build-system --- source/flow.rst | 32 ++++---------------------------- 1 file changed, 4 insertions(+), 28 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 0c56458a8..ca36a9c66 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -24,7 +24,7 @@ user's Python environment. The steps to achieve this are: will be a :file:`pyproject.toml` file, maintained maunally in the source tree. -- Create build artifacts to be sent to the package distribution service +- Create build artifacts to be sent to the package distribution service (usually PyPI); these will normally be a :term:`source distribution ("sdist") ` and one or more :term:`built distributions ("wheels") `. @@ -69,35 +69,13 @@ format`_. At a minimum, the :file:`pyproject.toml` file needs a ``[build-system]`` table specifying your build tool. There are many build tools available, including but not limited to :ref:`flit`, :ref:`hatch`, :ref:`pdm`, :ref:`poetry`, -:ref:`setuptools`, `trampolim`_, and `whey`_. +:ref:`setuptools`, `trampolim`_, and `whey`_. Each tool's documentation will +show what to put in the ``[build-system]`` table. .. _trampolim: https://pypi.org/project/trampolim/ .. _whey: https://pypi.org/project/whey/ -.. TODO: Only show Hatchling, or use tabs ala packaging tutorial? - -For example, here is a table for using ``setuptools`` (see the `Setuptools -documentation`_): - -.. code-block:: toml - - [build-system] - requires = ["setuptools>=61.2"] - build-backend = "setuptools.build_meta" - -.. _Setuptools documentation: https://setuptools.pypa.io/en/latest/userguide/index.html - -or for using ``flit`` (see the `Flit documentation`_): - -.. code-block:: toml - - [build-system] - requires = ["flit_core >=3.2,<4"] - build-backend = "flit_core.buildapi" - -.. _Flit documentation: https://flit.pypa.io/en/latest/ - -or for using :ref:`Hatchling ` (see the `Hatch documentation`_): +For example, here is a table for using :ref:`hatch`: .. code-block:: toml @@ -105,8 +83,6 @@ or for using :ref:`Hatchling ` (see the `Hatch documentation`_): requires = ["hatchling"] build-backend = "hatchling.build" -.. _Hatch documentation: https://hatch.pypa.io/latest/ - With such a table in the :file:`pyproject.toml` file, a "frontend" tool like :ref:`build` can run your chosen build tool's "backend" to create the build artifacts. Your build tool may also provide its own frontend. An install tool From bb737e80107d91be534b432b6516f6f087a0b036 Mon Sep 17 00:00:00 2001 From: Brian Rutledge Date: Thu, 28 Jul 2022 08:05:18 -0400 Subject: [PATCH 28/30] Always use ref for tools --- source/flow.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index ca36a9c66..3aeadbf62 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -86,8 +86,8 @@ For example, here is a table for using :ref:`hatch`: With such a table in the :file:`pyproject.toml` file, a "frontend" tool like :ref:`build` can run your chosen build tool's "backend" to create the build artifacts. Your build tool may also provide its own frontend. An install tool -like ``pip`` also acts as a frontend when it runs your build tool's backend to -install from a source distribution. +like :ref:`pip` also acts as a frontend when it runs your build tool's backend +to install from a source distribution. The particular build tool you choose dictates what additional information is required in the :file:`pyproject.toml` file. For example, you might specify: @@ -112,7 +112,7 @@ also include tests and documentation. These are useful for end users wanting to develop your sources, and for end user systems where some local compilation step is required (such as a C extension). -The ``build`` package knows how to invoke your build tool to create one of +The :ref:`build` package knows how to invoke your build tool to create one of these: .. code-block:: bash @@ -133,10 +133,10 @@ makes the install faster and more convenient for end users. A pure Python package typically needs only one "generic" wheel. A package with compiled binary extensions needs a wheel for each supported combination of Python interprerter, operating system, and CPU architecture that it supports. -If a suitable wheel file is not available, tools like ``pip`` will fall back -to installing the source distribtion. +If a suitable wheel file is not available, tools like :ref:`pip` will fall +back to installing the source distribtion. -The ``build`` package knows how to invoke your build tool to create one of +The :ref:`build` package knows how to invoke your build tool to create one of these: .. code-block:: bash @@ -147,9 +147,9 @@ Or, your build tool may provide its own interface for creating a wheel. .. note:: - The default behaviour of ``build`` is to make both an sdist and a wheel from - the source in the current directory; the above examples are deliberately - specific. + The default behaviour of :ref:`build` is to make both an sdist and a wheel + from the source in the current directory; the above examples are + deliberately specific. Upload to the package distribution service ========================================== @@ -167,7 +167,7 @@ Download and install ==================== Now that the package is published, end users can download and install the -package into their Python environment. Typically this is done with ``pip``, +package into their Python environment. Typically this is done with :ref:`pip`, using a command like: .. code-block:: bash From 68eb42e86de865927e92cefe098188de093e2424 Mon Sep 17 00:00:00 2001 From: Brian Rutledge Date: Thu, 28 Jul 2022 08:11:15 -0400 Subject: [PATCH 29/30] Link other install tools --- source/flow.rst | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/source/flow.rst b/source/flow.rst index 3aeadbf62..891a0d432 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -44,8 +44,6 @@ To use the package, end users must: directory. This step may involve a build/compile step which, if needed, must be described by the package metadata. -.. TODO: All other tools in the steps above are mentioned later - These last 2 steps are typically performed by a tool like :ref:`pip`. The steps above are described in more detail below. @@ -174,10 +172,5 @@ using a command like: python3 -m pip install package-name -.. TODO: Briefly describe typical behavior of using a virtual environment, -.. and maybe link to one or more of: -.. tutorials/installing-packages -.. guides/installing-using-pip-and-virtual-environments -.. guides/installing-stand-alone-command-line-tools.html - -.. TODO: Mention poetry, pdm, and pipenv as other methods? \ No newline at end of file +End users may also use other tools like :ref:`pipenv`, :ref:`poetry`, or +:ref:`pdm`. From 0e08f5fc7199478abec55e3376cd953eb166d299 Mon Sep 17 00:00:00 2001 From: Brian Rutledge Date: Thu, 28 Jul 2022 18:32:05 -0400 Subject: [PATCH 30/30] Tweak pip reference --- source/flow.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/source/flow.rst b/source/flow.rst index 891a0d432..d1d82b190 100644 --- a/source/flow.rst +++ b/source/flow.rst @@ -44,7 +44,8 @@ To use the package, end users must: directory. This step may involve a build/compile step which, if needed, must be described by the package metadata. -These last 2 steps are typically performed by a tool like :ref:`pip`. +These last 2 steps are typically performed by :ref:`pip` when an end user runs +``pip install``. The steps above are described in more detail below.