diff --git a/images/python-package-dependency-types.png b/images/python-package-dependency-types.png index 6555b198..d2a67f41 100644 Binary files a/images/python-package-dependency-types.png and b/images/python-package-dependency-types.png differ diff --git a/package-structure-code/declare-dependencies.md b/package-structure-code/declare-dependencies.md index e498d506..2d43c792 100644 --- a/package-structure-code/declare-dependencies.md +++ b/package-structure-code/declare-dependencies.md @@ -1,377 +1,527 @@ ```{eval-rst} -:og:title: Add required and optional dependencies to your Python package +:og:title: Add Required, Optional Dependencies and Dependency Groups to Your Python Package :og:description: A Python package dependency refers to an external package or software that your Python project requires to function properly. Learn how to add different types of dependencies to your Python package. ``` +# Dependencies for your Python Package +In the [pyproject.toml overview page](pyproject-toml-python-package-metadata), +you learned how to set up a **pyproject.toml** file with basic metadata +for your package. On this page, you will learn how to specify different types of +dependencies in your `pyproject.toml`. -:::{todo} +## What is a package dependency? -keep this comment - https://github.com/pyOpenSci/python-package-guide/pull/106#issuecomment-1844278487 in this file for now - jeremiah did a nice inventory of common shells and whether they need quotes or not. It's really comprehensive. But do we want it in the guide?? it's really useful for more advanced users i think. +A Python package dependency refers to an external package or +A tool that is needed when using or working on your Python project. Declare your dependencies in your `pyproject.toml` file. This keeps all package metadata in one place, making it simpler for users and contributors to understand your package. -Following this comment: -https://github.com/pyOpenSci/python-package-guide/pull/106#pullrequestreview-1766663571 +:::{admonition} Older ways to declare dependencies +:class: tip -Jonny will add a section that talks about: +While `pyproject.toml` is now the standard, you may sometimes encounter older approaches to storing dependencies "in the wild": +- **requirements.txt**: Previously common for dependencies, still used by some projects for local development +- **setup.py or setup.cfg**: May be needed for packages with extensions in other languages -Why you specify dependencies -How to specify dependencies -When you use different specifiers +[Learn more in the setuptools documentation](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#declaring-required-dependency) ::: -# Python Package Dependencies - -## What is a package dependency? - -A Python package dependency refers to an external package or -software that your Python project: +### Why specify dependencies -1. Needs to function properly. -2. Requires if someone wants to develop / work on improving your package locally or -3. Requires if a user wants to add additional functionality (that is not core) to your package +Specifying dependencies in the [project.dependency] array of your `pyproject.toml` file ensures that libraries needed to run your package are correctly installed into a user's environment. +For instance, if your package requires Pandas to run properly, and you add Pandas to the `project.dependency` array, Pandas will be installed into the users' environment when they install your package using uv, pip, or conda. -A dependency is not part of your project's codebase. It is a package or software that is called -within the code of your project or during development of your package. +```toml +[project] +... +... +... +dependencies = [ + "pandas", +] +``` +Development dependencies make it easier for contributors to work on your package. You can set up instructions for running specific workflows, such as tests, linting, and even typing, that automatically install groups of development dependencies. These dependencies can be stored in arrays (lists of dependencies) within a `[development-group]` table. -### Understanding optional vs. required dependencies -You can think about dependencies as being either optional or required. If they are required, they will be listed in the `dependencies` key in the `project` table of your `pyproject.toml` file. If they are optional, they will be listed in the `[optional.dependencies]` table of your `pyproject.toml`. +```toml +[development-group] +tests = [ + "pytest", + "pytest-cov" +] +``` -You will learn about both below. +### Types of dependencies -:::{figure-md} python-package-dependency-types +There are three different types of dependencies that you will learn about on this page: - +1. **Required dependencies:** These are dependencies that need to be installed for your package to work correctly in a user's environment. You add these dependencies to the `[project.dependencies]` table in your pyproject.toml file. +2. **Feature Dependencies:** These are dependencies that are required if a user wants to access additional functionality (that is not core) to your package. Store these in the `[project.optional.dependencies]` table or your pyproject.toml file. +3. **Development Dependencies:** These dependencies are required if someone wants to develop or work on your package. These include instance linters, testing tools like pytest and mypy are examples of development dependencies. Store these in the `[project.dependency.groups]` table or your pyproject.toml file. -There are two broad groups of Python package dependencies: those that are optional and those that are required. Required packages are those that a user needs to use your package. Optional dependencies are packages a user can chose to install to add functionality to your package. -Within those 2 groups, there are three use cases that you can think about. 1. Core dependencies are **required** for a user to use your package. 2. Development dependencies are optional and only needed if someone wants to work on your package locally. 3. Finally feature dependencies are optional and add additional functionality to your package. Not all packages will have feature dependencies. +:::{admonition} +A dependency is not part of your project's codebase. It is a package or software called +within the code of your project or used during the development of your package. ::: +## 1. Required dependencies -### Required (or core) dependencies +Required dependencies are imported and called directly within your package's code. +They are needed for your package to run. -Required dependencies are called directly within your package's code. On this page we refer to these dependencies -as **core dependencies** as they are needed in order to run your package. You should place your core or required dependencies in the `dependencies` key of the `[project]` table of your `pyproject.toml` file. +You can add your required dependencies to the `dependencies` array in the +`[project]` table of your **pyproject.toml** file. When users install +your package with uv, pip, or conda, these dependencies will be +automatically installed alongside your package in their environment. -### Optional dependencies +```toml +[project] +name = "examplePy" +authors = [ + {name = "Some Maintainer", email = "some-email@pyopensci.org"}, +] +dependencies = [ + "pandas", + "matplotlib", +] +``` -Optional dependencies dependencies can be optionally installed by users -depending upon their needs. There are two broad groups of optional dependencies: +:::{admonition +Try your best to minimize dependencies whenever possible. Remember that +fewer dependencies reduce the possibility of version conflicts in user +environments. +::: -1. **Development dependencies**: These are dependencies that are required to support development of your package. They include tools to run tests such as `pytest`, linters (like `flake8` and `ruff`) and code formatters such as `black` and even automation tools such as `nox` or `tox` that run tasks. -2. **Feature dependencies:** These are dependencies that a user can chose to install to add functionality to your package. +::::{dropdown} How to Add Required Dependencies with UV +:icon: eye +:color: primary -When a Python project is installed, the Python package manager (either `pip` -or `conda`) installs your package's dependencies automatically. This ensures -that when you call a function in a specific dependency, it is available in your -user's environment. +You can use uv to add dependencies to your pyproject.toml file: -:::{admonition} Dependencies can be added to your pyproject.toml file +**Add a required dependency:** -In the [pyproject.toml overview page](pyproject-toml-python-package-metadata), -you learned how to set up a **pyproject.toml** file with basic metadata -for your package. On this page, you will learn how to specify different types of -dependencies in your `pyproject.toml`. +```bash +$ uv add numpy +``` -::: +Will add numpy as a dependency to your `project.dependency` array: -## How do you declare dependencies? +```toml +[project] -We recommend that you declare your dependencies using your `pyproject.toml` file. -This ensures that all of the metadata associated with your package is declared -in a single place, making it simpler for users and contributors to understand -your package infrastructure. +dependencies = [ + "numpy>=2.2.6", +] +``` -Previously, it was common to use a `requirements.txt` file to declare package dependencies. -However in recent years, the ecosystem has moved to storing this -information in your **pyproject.toml** file. You may notice however that some -projects still maintain a `requirements.txt` file for specific local development -needs. +:::: -:::{admonition} Other ways you may see packages storing dependencies +:::{admonition} Requiring packages from GitHub / Gitlab :class: tip -If a project contains extensions written in other languages, you may need a `setup.py` file. Or you may contribute to a package that us using `setup.cfg` for dependency declaration. -[Learn more about this in the setuptools documentation](https://setuptools.pypa.io/en/latest/userguide/dependency_management.html#declaring-required-dependency) -::: - - -### Add required dependencies to your pyproject.toml file - -Your core project dependencies need to be installed by a -package manager such as `pip` or `conda` when a user installs your package. You can add those dependencies to -the -`dependencies` array located within the `[project]` table of your -**pyproject.toml** file. This looks something like this: +If you have dependencies that need to be installed directly from GitHub, +you can specify them in your pyproject.toml file like this: ```toml -[project] -name = "examplePy" -authors = [ - {name = "Some Maintainer", email = "some-email@pyopensci.org"}, -] - dependencies = [ - "rioxarray", - "geopandas", +"my_dependency >= 1.0.1 @ git+https://git.server.example.com/mydependency.git@commitHashHere" ] ``` +IMPORTANT: If your library depends on a GitHub-hosted project, +you should point to a specific commit/tag/hash of that repository before you upload your project to +PyPI. You never know how the project might change over time. Commit hashes +are more reliable as they can't be changed +::: -Ideally, you should only list the packages that are -necessary to install and use your package in the -`dependencies` key in the `[project]` table. This minimizes the number of -additional packages that your users must install as well -as the number of packages that depend upon your package -must also install. - -Remember that fewer dependencies to install reduces the -likelihood of version mismatches in user environments. - -:::{admonition} A dependency example +## 2. Optional dependencies -Let's pretend you have a package called `plotMe` that creates beautiful plots of data stored in `numpy` arrays. To create your plots in the `plotMe` package, you use the `seaborn` package to stylize our plots and also `numpy` to process array formatted data. +Optional (also referred to as feature) dependencies can be installed by users as needed. Optional dependencies add specific features to your package that not all users need. For example, if your package has an optional interactive plotting feature that uses Bokeh, you would list Bokeh as an `[optional.dependency]`. Users who want interactive plotting will install it. Users who don't need plotting don't have to install it. -In the example above, the plotMe package, depends upon two packages: +Place these dependencies in the `[project.optional-dependencies]` table. -* seaborn -* numpy +```toml +[project] +... +... +... +[optional.dependencies] +plot = ["bokeh"] +``` -This means that in order for plotMe to work in a user's `environment` when installed, you also need to ensure that they have both of those required `dependencies` installed in their environment too. +When a user installs your package, uv, pip, or conda automatically installs all required dependencies. Optional dependencies are only installed if the user explicitly requests them. -Declaring a dependency in your `pyproject.toml` file will ensure that it is listed as a required dependency when your package is published to PyPI and that a package manager (`pip` or `conda`) will automatically install it into a user's environment alongside your package: +:::{dropdown} How to Add optional.dependencies using UV +:icon: eye +:color: primary -`python -m pip install plotMe` -::: +You can use uv to add dependencies to your pyproject.toml file: -### Optional dependencies +**Add an optional dependency:** -Optional dependencies for building your documentation, running your tests and building your package's distribution files are often referred to as development dependencies. These are the dependencies that a user needs to work on your package locally and perform tasks such as: +```bash +uv add --optional feature pandas +``` -* running your test suite -* building your documentation -* linting and other code cleanup tools +Will add this to your pyproject.toml file: +```toml +[optional.dependencies] +feature = [ + "pandas>=2.3.3", +] +``` +::: -These dependencies are considered optional, because they are not required to install and use your package. Feature -dependencies are considered optional and should also be placed in the `[project.optional-dependencies]` table. +## 3. Dependency groups -Optional dependencies can be stored in an -`[project.optional-dependencies]` table in your **pyproject.toml** file. +Development dependencies include packages needed to work on your package +locally. They are used to perform tasks such as: -It's important to note that within the `[project.optional-dependencies]` table, you can store additional, optional dependencies within named sub-groups. This is a different table than the dependencies array located within the `[project]` table discussed above which contains a single array with a single list of required packages. +* running your test suite (pytest, pytest-cov) +* building your documentation (sphinx, sphinx-theme packages) +* linting and formatting code (ruff, black) +* building package distribution files (build, twine) -## Create optional dependency groups +Dependency groups are optional because they are not required for users +to install and use your package. However, they will make it easier for +contributors to your project to setup development environments +locally. -To declare optional dependencies in your **pyproject.toml** file: +:::{admonition} New: PEP 735 development dependency groups +:class: note -1. Add a `[project.optional-dependencies]` table to your **pyproject.toml** file. -2. Create named groups of dependencies using the syntax: +`[development-groups]` is a newer specification introduced by PEP 735. +They are intended to organize development dependencies and are intentionally separate from `[project.optional-dependencies]`, which can be installed into a user's +environment. +::: -`group-name = ["dep1", "dep2"]` +### How to declare dependency groups -:::{admonition} Installing packages from GitHub / Gitlab -:class: tip +You declare development dependencies in your **pyproject.toml** file +within a `[development-groups]` table. -If you have dependencies that need to be installed directly from GitHub using -a `git+https` installation approach, you can do so using the pyproject.toml -file like so: +Similar to optional-dependencies, you can create separate subgroups or arrays with names using the syntax: `group-name = ["dep1", "dep2"]` ```toml -dependencies = [ -"my_dependency >= 1.0.1 @ git+https://git.server.example.com/mydependency.git" -] +[development-groups] +tests = ["pytest", "pytest-cov"] +docs = ["sphinx", "pydata-sphinx-theme"] +lint = ["ruff", "black"] ``` -IMPORTANT: For security reasons, if your library depends on a GitHub-hosted -project, you will need to point to a specific commit/tag/hash of that repository in -order to upload your project to PyPI -::: +:::{dropdown} How to Add [development.group] using UV +:icon: eye +:color: primary + +You can use uv to add dependencies to your pyproject.toml file: -Below we've created three sets of optional development dependencies named: tests, docs and lint. We've also added a set of feature dependencies. +**Add a development group dependency:** +```bash +uv add --group tests pytest +uv add --group docs sphinx +``` +Will add the following to your pyproject.toml file: ```toml -[project.optional-dependencies] +[dependency-groups] tests = [ - "pytest", - "pytest-cov" + "pytest>=8.4.2", ] docs = [ - "sphinx", - "pydata_sphinx_theme" -] -lint = [ - "black", - "flake8" -] -feature = [ - "pandas", + "sphinx>=8.1.3", ] - ``` +::: + +:::{todo} +i'll pick back up here tomorrow - this section is all about how things install and what "ships" with your package vs what just gets installed via commands (ie development) + +::: +## Understanding required vs. optional dependencies + +:::{todo} +The purpose of this section is to help users understand how dependencies relate to what is installed in their environment. We have two graphics on this page - one that breaks out the two buckets of tools (required and optional) that both get installed into a user's envt vs development groups, which are contributor/ development facing, not user-facing. When we originally wrote this section, development groups didn't exist, and we were using optional dependencies for dev groups. + +The graphic below is two circles representing optional vs regular / required deps - created before development groups existed... there is another graphi that shows what gets installed into a uses envt. +::: + +:::{figure-md} python-package-dependency-types +Diagram showing two main groups of Python package dependencies: required and optional. Required dependencies include core packages needed to use your package. Optional dependencies include development dependencies for working on the package locally and feature dependencies for additional functionality. + +Python package dependencies fall into two categories: **required** +dependencies that users need to run your package, and **optional** +dependencies for development work or additional features. +::: + + :::{admonition} Additional dependency resources * [Learn more: View PyPA's overview of declaring optional dependencies](https://packaging.python.org/en/latest/specifications/declaring-project-metadata/#dependencies-optional-dependencies) - * [Dependency specifiers](https://packaging.python.org/en/latest/specifications/dependency-specifiers/) ::: ### Install dependency groups +When someone installs your package, only core dependencies are installed by +default. To install optional dependencies, you +need to specify which groups to include when installing the package. :::{figure-md} python-package-dependencies -Diagram showing a Venn diagram with three sections representing the dependency groups listed above - docs feature and tests. In the center it says your-package and lists the core dependencies of that package seaborn and numpy. To the right are two arrows. The first shows the command python - m pip install your-package. It them shows how installing your package that way installs only the package and the two core dependencies into a users environment. Below is a second arrow with python -m pip install youPackage[tests]. This leads to an environment with both the package dependencies - your-package, seaborn and numpy and also the tests dependencies including pytest and pytest-cov +Diagram showing a Venn diagram with three sections representing dependency groups - docs, feature, and tests. In the center it shows your-package with core dependencies seaborn and numpy. Two arrows on the right demonstrate: first, python -m pip install your-package installs only the package and core dependencies. Second, python -m pip install your-package[tests] installs the package, core dependencies, and test dependencies including pytest and pytest-cov. -When a user installs your package locally using `python -m pip install your-package` only your package and it's core dependencies get installed. When they install your package `python -m pip install your-package[tests]` pip will install both your package and its core dependencies plus any of the dependencies listed within the tests array of your `[project.optional-dependencies]` table. +When a user installs your package using `pip install your-package`, only +your package and its core dependencies get installed. When they install +with `pip install your-package[tests]`, pip will install your package, +core dependencies, and the test dependencies from the +`[project.optional-dependencies]` table. ::: -:::{admonition} Using `python -m pip install` vs. `pip install` +### Using uv or pip for installation -In all of the examples in this guide, you will notice we are calling -`pip` using the syntax: +UV streamlines this process, allowing you to sync a venv in your project directory +with both an editable install of your package and its dependencies automatically. +You can also use pip and install dependencies into the environment of your choice. -`python -m pip` +:::{todo} +We shouldn't show UV pip install, so how do you add optional feature deps with UV?? +::: -Calling pip using `python -m` ensures that the `pip` that you are using to install your package comes from your current active Python -environment. We strongly suggest that you use this approach whenever -you call `pip` to avoid installation conflicts. -To ensure this works as you want it to, activate your package's development -environment prior to installing anything using `pip`. -::: +**Install development groups:** -You can install development dependencies using the -groups that you defined above using the syntax: +:::::{tab-set} -`python -m pip install ".[docs]"` +::::{tab-item} Use UV +You can use uv sync to sync dependency groups in your uv-managed venv -Above you install: -* dependencies needed for your documentation (`docs`), -* required package dependencies in the `dependencies` array and -* your package +```console +$ uv sync --group docs # Single group +$ uv sync --group docs --group test # Multiple groups +$ uv sync --all-groups # All development groups +``` -using pip. Below you -install your package, required dependencies and optional test dependencies. +**Install optional dependencies:** -`python -m pip install ".[tests]"` +```console +# uv pip install is not idea if you are using uv supported venvs for your project +$ uv pip install -e ".[docs]" # Single group +$ uv pip install -e ".[docs,tests,lint]" # Multiple groups +``` -You can install multiple dependency groups in the `[project.optional-dependencies]` table using: +**Install everything (package + all dependencies):** -`python -m pip install ".[docs, tests, lint]"` +```console +$ uv sync --all-extras --all-groups +``` +`uv sync` is the recommended command for development workflows. It +manages your virtual environment and keeps your lockfile up to date. +Use `uv pip install` when you need pip-compatible behavior. +:::: -```{admonition} For zsh shell users -:class: tip +::::{tab-item} Use pip (version >=25.1) -There are different shell applications that you and your package contributors might use. -* zsh is the shell that comes by default on newer Mac OS computers -* Windows users may use a tool such as git bash +**Install optional dependencies:** -Some shells don't support unquoted brackets (`[tests]`) which is why we add -quotes to the command in this guide like this: +```console +python -m pip install -e ".[docs]" # Single group +python -m pip install -e ".[docs,tests,lint]" # Multiple groups +``` +**Install dependency groups:** + +```console +python -m pip install --group test # Single group +python -m pip install --group docs # Multiple groups +``` + +Always call pip using `python -m pip` to ensure you're using +the pip from your current active Python environment. This helps avoid +installation conflicts. + +**Note:** Some shells (like zsh on Mac) require quotes around brackets to run successfully: `python -m pip install ".[tests]"` -In some cases you may see commands without the quotes in guidebooks or contributing -guides like the example below: +::: +:::: -`python -m pip install your-package[tests]` +::::: -Calling your-package[tests] without the double quotes will work on some shells *but not all*. -``` -### Combining sets of dependencies +### Combining dependency groups -Above we reviewed how to install dependencies from your `pyproject.toml`. In some cases you may want to group sets of dependencies like so: +You can also create combined groups that reference other groups: ```toml [project.optional-dependencies] -tests = ["pytest", "pytest-cov"] -docs = ["sphinx", "pydata_sphinx_theme"] -dev = [ - "packageName[tests, docs]", - "build", - "twine" -] +test = ["pytest", "pytest-cov"] +docs = ["sphinx", "pydata-sphinx-theme"] +dev = ["your-package[test,docs]", "build", "twine"] +``` + +Then install everything with pip install or uv sync as needed: +```bash +uv pip install -e ".[dev]" +# or +python -m pip install ".[dev]" ``` -The above allows you to install both the tests and docs dependency lists -using the command: +:::{tip} +When you install optional dependencies, pip and uv install your +package and its core dependencies automatically. +::: + -`python -m pip install ".[dev]"` +## Version specifiers for dependencies -```{tip} -When you install dependencies using the above syntax: +Version specifiers control which versions of a dependency work with your +package. Use them to specify minimum versions, exclude buggy releases, or +set version ranges. -`python -m pip install ".[tests, docs]"` +### Common operators -`pip` will also install your package and its core dependencies. +- **`>=`** Minimum version set: `numpy>=1.20` (This is the most common approach and is recommended) +- **`==`** Exact version: `requests==2.28.0` (Avoid pinning dependencies like this unless necessary) +- **`~=`** Compatible release: `django~=4.2.0` (Allows patches: >=4.2.0,<4.3.0) +- **`<` or `>`** - Upper/lower bounds: `pandas>=1.0,<3.0` +- **`!=`** Exclude version: `scipy>=1.7,!=1.8.0` (Rare but allows you to skip a buggy release version) + +:::{tip} +**Best practice:** Use `>=` to specify your minimum tested version and +avoid upper bounds unless you know at what version that dependency is no longer compatible. UV will do this by +default when it adds a dependency to your pyproject.toml file. This keeps +your package flexible and reduces dependency conflicts. +```toml +dependencies = [ + "numpy>=1.20", # Good - flexible + "pandas>=1.0,<3.0", # OK - known breaking change in 3.0 + "requests==2.28.0", # Avoid - too restrictive +] ``` +::: -:::{admonition} Where does conda fit in? -:class: note +:::{todo} +### Using conda and pixi + +::::{todo} +Ask Matthew to review this section... +::: + +The `pyproject.toml` file works great for pure-Python packages. However, +some packages (particularly in the scientific Python ecosystem) require +dependencies written in other languages like C or Fortran. Conda was +created to support the distribution of tools with non-Python dependencies. + +**For conda users:** + +You can maintain an `environment.yml` file to help users and contributors +set up conda environments. This is especially useful for packages with +system-level dependencies like GDAL. + +**Consider pixi for conda workflows:** + +[Pixi](https://pixi.sh) is a modern package manager built on conda that +uses `pyproject.toml` for configuration. If your project relies heavily +on conda packages, pixi offers a streamlined workflow with lockfiles and +faster dependency resolution. + +:::{admonition} A note for conda users +:class: tip + +If you use a conda environment for development and install your package +with `python -m pip install -e .`, dependencies will be installed from +PyPI rather than conda. This can cause conflicts, especially for packages +with system dependencies. -The `pyproject.toml` file allows you to list any -Python package published on PyPI (or on GitHub/ GitLab) as a dependency. Once you create this file, declare dependencies, [build your package](python-package-distribution-files-sdist-wheel.md) and [publish your package to PyPI](publish-python-package-pypi-conda.md), people can install both your package and all of it's dependencies with one command. +To avoid this, install your package without dependencies: +`python -m pip install -e . --no-deps` -`python -m pip install your-package` +Then install dependencies through your conda environment.yml file. +::: +:::: -This works great if your package is pure-python (no other languages used). +### Using conda and pixi -Some packages, particularly in the scientific Python ecosystem, require dependencies that are not written in Python. Conda was created to support distribution of tools that have code written in both Python and languages other than Python. +:::{todo} +Ask matthew to review this section... ::: -## Support conda users with environment.yml files +The `pyproject.toml` file works great for pure-Python packages. However, +some packages (particularly in the scientific Python ecosystem) require +dependencies written in other languages like C or Fortran. Conda was +created to support the distribution of tools with non-Python dependencies. + +**For conda users:** -The above workflow assumes that you want to publish your package on PyPI. And then you plan to publish to conda-forge (optionally), [by submitting a recipe using grayskull](https://www.pyopensci.org/python-package-guide/package-structure-code/publish-python-package-pypi-conda.html). +You can maintain an `environment.yml` file to help users and contributors +set up conda environments. This is especially useful for packages with +system-level dependencies like GDAL. -If you want to support conda users, you may want to also maintain a conda environment that they can use to install your package. Maintaining a conda environment will also help you test that your package installs as you expect into a conda environment. +**Consider pixi for conda workflows:** +[Pixi](https://pixi.sh) is a modern package manager built on conda that +uses `pyproject.toml` for configuration. If your project relies heavily +on conda packages, pixi offers a streamlined workflow with lockfiles and +faster dependency resolution. -```{admonition} A note for conda users +:::{admonition} A note for conda users :class: tip -If you use a conda environment for developing your tool, keep in mind that when you install your package using `python -m pip install -e .` (or using pip in general), dependencies will be installed from PyPI rather than conda. +If you use a conda environment for development and install your package +with `python -m pip install -e .`, dependencies will be installed from +PyPI rather than conda. This can cause conflicts, especially for packages +with system dependencies. -Thus, if you are running a conda environment, installing your package in "editable" mode risks dependency conflicts. This is particularly important if you have a spatial package that requires geospatial system libraries like GDAL or another system-level dependency. +To avoid this, install your package without dependencies: +`python -m pip install -e . --no-deps` -Alternatively, you can install your package using `python -m pip install -e . --no-deps` to only install the package. And install the rest of your dependencies using a conda environment file. -``` +Then install dependencies through your conda environment.yml file. +::: ## Dependencies in Read the Docs -Now that you have your dependencies specified in your project, you can use them to support other workflows such as publishing to Read the Docs. +Once you've specified dependencies in your `pyproject.toml`, you can use +them in other workflows like building documentation on Read the Docs. -[Read the Docs](https://readthedocs.org) is a documentation platform with a continuous integration / continuous deployment service that automatically builds and publishes your documentation. - -If you are using Read the Docs to build your documentation, then you may need to install your dependencies using a **readthedocs.yaml** file. - -Below is an example of installing the **docs** section of your dependency table in the pyproject.toml file within a readthedocs.yaml file. +[Read the Docs](https://readthedocs.org) is a documentation platform +that automatically builds and publishes your documentation. To install +your dependencies during the build process, configure them in a +**readthedocs.yaml** file. +Here's an example that installs your `docs` optional dependencies: ```yaml python: install: - method: pip path: . extra_requirements: - - docs # you can add any of the subgroups of dependencies from your pyproject.toml file to this list. + - docs ``` - -:::{admonition} Read the Docs and Python packages +:::{admonition} Learn more about Read the Docs :class: note -* [Learn more about creating a `readthedocs.yaml` file here. ](https://docs.readthedocs.io/en/stable/config-file/index.html) -* If you want to install dependencies using -Poetry in Read the Docs, [you can learn more here.](https://docs.readthedocs.io/en/stable/build-customization.html#install-dependencies-with-poetry) - +* [Creating a readthedocs.yaml file](https://docs.readthedocs.io/en/stable/config-file/index.html) +* [Using uv with Read the Docs](https://docs.readthedocs.io/en/stable/build-customization.html) +* [Using Poetry with Read the Docs](https://docs.readthedocs.io/en/stable/build-customization.html#install-dependencies-with-poetry) ::: + :::{todo} -This is hidden. TO + +Keep this comment - https://github.com/pyOpenSci/python-package-guide/pull/106#issuecomment-1844278487 in this file for now - Jeremiah did a nice inventory of common shells and whether they need quotes or not. It's really comprehensive. But do we want it in the guide?? It's really useful for more advanced users. + +Following this comment: +https://github.com/pyOpenSci/python-package-guide/pull/106#pullrequestreview-1766663571 + +Jonny will add a section that talks about: + +Why you specify dependencies +How to specify dependencies +When you use different specifiers ::: diff --git a/package-structure-code/pyproject-toml-python-package-metadata.md b/package-structure-code/pyproject-toml-python-package-metadata.md index 937b743c..9eb2317e 100644 --- a/package-structure-code/pyproject-toml-python-package-metadata.md +++ b/package-structure-code/pyproject-toml-python-package-metadata.md @@ -1,22 +1,13 @@ (pyprojecttoml-metadata)= # Use a pyproject.toml file for your package configuration & metadata - - -:::{admonition} Important pyproject.toml take aways -:class: todo +:::{admonition} pyproject.toml takeaways 1. There are only two tables that are required for an installable Python package: **[build-system]** and **[project]**. The **[project]** table stores your package's metadata. -2. There are only two _required_ fields in the **[project]** table: **name=** and **version=**. -3. We suggest you add additional metadata to your `pyproject.toml` file as it will make it easier for users to find your project on PyPI. +2. There are two _required_ fields in the **[project]** table: **name=** and **version=**. +3. Add metadata to the classifiers section of your `pyproject.toml` file to make it easier for users to find your project on PyPI. 4. When you are adding classifiers to the [project] table, only use valid values from [PyPI’s classifier page](https://PyPI.org/classifiers/). An invalid value here will raise an error when you build your package or publish to PyPI. 5. There is no specific order for tables in the `pyproject.toml` file. However fields need to be placed within the correct table sections. For example `requires =` always need to be associated with the **[build-system]** table. -6. **python-requires**: is important to have in your `pyproject.toml` file as it helps pip install your package. ::: @@ -57,36 +48,26 @@ Click here if need help migrating from setup.py/setup.cfg to pyproject.toml ## About the pyproject.toml file -Every modern Python package should include a `pyproject.toml` file. If your project is pure Python and you're using a `setup.py` or `setup.cfg` file to describe its metadata, you should consider migrating your metadata and build information to a `pyproject.toml` file. +Every modern Python package should include a `pyproject.toml` file. For pure Python packages, this file replaces the `setup.py` and/or `setup.cfg` file to describe project metadata. -If your project isn’t pure-python, you might still require a `setup.py` file to build the non Python extensions. However, a `pyproject.toml` file should still be used to store your project’s metadata. +If your project isn’t pure Python, you might still require a `setup.py` file to build the non-Python extensions. However, a `pyproject.toml` file should still be used to store your project’s metadata. -:::{admonition} What happened to setup.py & how do i migrate to pyproject.toml? +:::{admonition} Tutorial :class: note -Prior to August 2017, Python package metadata was stored either in the `setup.py` file or a `setup.cfg` file. In recent years, there has been a shift to storing Python package metadata in a much more user-readable `pyproject.toml` format. Having all metadata in a single file: -- simplifies package management, -- allows you to use a suite of different [build backends](https://www.pyopensci.org/python-package-guide/package-structure-code/python-package-build-tools.html#build-back-ends) such as (flit-core, hatchling, pdm-build), and -- aligns with modern best practices. - - +If you are migrating from a **setup.py** or **setup.cfg** file, and want help, [check out this tutorial.](migrate-pyproj) +[specify build requirements and +metadata is called a **pyproject.toml**](https://packaging.python.org/en/latest/specifications/declaring-project-metadata/) ::: -The standard file that Python packages use to [specify build requirements and -metadata is called a **pyproject.toml**](https://packaging.python.org/en/latest/specifications/declaring-project-metadata/). Adding metadata, build requirements -and package dependencies to a **pyproject.toml** file replaces storing that -information in a setup.py or setup.cfg file. - ### About the .toml format -The **pyproject.toml** file is written in [TOML (Tom's Obvious, Minimal Language) format](https://toml.io/en/). TOML is an easy-to-read structure that is founded on key/value pairs. Each section in the **pyproject.toml** file contains a `[table identifier]`. +The **pyproject.toml** file is written in [TOML (Tom's Obvious, Minimal Language) format](https://toml.io/en/). TOML is an easy-to-read structure based on key/value pairs. Each section in the **pyproject.toml** file contains a `[table identifier]`. Below that table identifier are key/value pairs that support configuration for that particular table. - Below `[build-system]` is considered a table in the toml language. -- Within the `build-system` table below `requires =` is a key. +- Within the `build-system` table, `requires =` is a key. - The associated value for `requires` is an array containing the value `"hatchling"`. :::{literalinclude} ../examples/pure-hatch/pyproject.toml @@ -116,12 +97,7 @@ represent on your PyPI landing page. These classifiers also allow users to sort Including your package's metadata in a separate human-readable **pyproject.toml** format also allows someone to view the project's metadata in a GitHub repository. - - -```{admonition} Setup.py is still useful for complex package builds +:::{admonition} Setup.py is still useful for complex package builds :class: tip Using **setup.py** to manage package builds and metadata [can cause problems with package development](https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html). @@ -129,7 +105,7 @@ In some cases where a Python package build is complex, a **setup.py** file may be required. While this guide will not cover complex builds, we will provide resources working with complex builds in the future. -``` +::: ## Optional vs. Required pyproject.toml file fields @@ -164,12 +140,12 @@ what dependencies your package requires. :end-at: ] ::: -- **dependencies:** dependencies are optional but we strongly suggest you include them in your pyproject.toml. Dependencies will be installed by pip when your project is installed creating a better user-experience. - -- **`[project.optional-dependencies]`:** the optional or development dependencies will be installed if someone runs `python -m pip install projectname[dev]`. This is a nice way to include your development dependencies for users who may wish to contribute to your project. +- **project.dependencies:** The dependency group is optional because not all packages require dependencies. However, if your project has specific dependencies, include this section in your `pyproject.toml`. Dependencies declared in the pyproject.toml file will be installed by uv or pip when your project is installed. +- **project.optional-dependencies:** Optional or feature dependencies will be installed if someone runs `python -m pip install projectname[feature]`. Use this array to declare dependencies that add specific features to your package that are not installed by default when a user runs `uv sync` or `python -m pip install packagename`. +- **dependency-groups:** Dependency groups organize packages and tools that a contributor or developer would need to work on your package. These dependencies may include tools for building and running tests, linters, and code formatters. This is an optional but highly suggested way to organize and install dependencies. This section can replace a requirements.txt file. [Learn more about adding these to your package in the PyPA guide here.](https://packaging.python.org/en/latest/specifications/dependency-groups/) - **keywords:** These are the keywords that will appear on your PyPI landing page. Think of them as words that people might use to search for your package. -- **classifiers:** The classifiers section of your metadata is also important for the landing page of your package in PyPI and for filtering of packages in PyPI. A list of [all options for classifiers can be found her](https://PyPI.org/classifiers/)e. Some of the classifiers that you should consider including +- **classifiers:** The classifiers section of your metadata is also important for the landing page of your package in PyPI and for filtering of packages in PyPI. A list of [all options for classifiers can be found here](https://PyPI.org/classifiers/). Some of the classifiers that you should consider including - Development Status - Intended Audience - Topic @@ -177,18 +153,16 @@ what dependencies your package requires. ### Advanced options in the pyproject.toml file -The examples at the bottom of this page contain ... - - **`[project.scripts]` (Entry points):** Entry points are optional. If you have a command line tool that runs a specific script hosted in your package, you may include an entry point to call that script directly at the command line (rather than at the Python shell). - Here is an example of[a package that has entry point script](https://github.com/pyOpenSci/pyosMeta/blob/main/pyproject.toml#L60)s. Notice that there are several core scripts defined in that package that perform sets of tasks. The pyOpenSci is using those scripts to process their metadata. -- **Dynamic Fields:** if you have fields that are dynamically populated. One example of this is if you are using scm / version control based version with tools like `setuptooms_scm`, then you might use the dynamic field, such as version (using scm) **dynamic = ["version"]** +- Use **Dynamic Fields** If you have fields that are dynamically populated. For example, you may wish to automatically update your package's version using Git tags (SCM/version control-based versioning). Example: **dynamic = ["version"]** ## Add dependencies to your pyproject.toml file -The pyproject.toml file can also be used as a replacement for the requirements.txt file which has been traditionally used to store development dependencies such as pytest, code formatters such as Black and documentation tools such as sphinx. +The `pyproject.toml` file is a modern replacement for the `requirements.txt` file, which has been traditionally used to store development dependencies and also configuration for tools such as pytest, black, and others. -To add dependencies to your build, add a `[project.optional-dependencies]` table to your pyproject.toml file. +To add development dependencies to your build, add a `[dependency-groups]` array to your pyproject.toml file. Then specify dependency groups as follows: @@ -202,6 +176,8 @@ Following the above example, you install dependencies like this: - `python -m pip install -e .[tests]` +- pip install --group test *# requires pip 25.1 or greater* + The above will install both your package in editable mode and all of the dependencies declared in the tests section of your `[project.optional-dependencies]` table. To install all dependencies and also your package, you'd use: diff --git a/tutorials/setup-py-to-pyproject-toml.md b/tutorials/setup-py-to-pyproject-toml.md index 8ab090b8..0b595cc7 100644 --- a/tutorials/setup-py-to-pyproject-toml.md +++ b/tutorials/setup-py-to-pyproject-toml.md @@ -3,6 +3,7 @@ :og:title: Using Hatch to Migrate setup.py to a pyproject.toml --- +(migrate-pyproj)= # Using Hatch to Migrate setup.py to a pyproject.toml Hatch can be useful for generating your project's `pyproject.toml` file if your project already has a `setup.py` file.