pip-tools = pip-compile + pip-sync
A set of command line tools to help you keep your
pip-based packages fresh,
even when you've pinned them. You do pin them, right? (In building your Python application and its dependencies for production, you want to make sure that your builds are predictable and deterministic.)
pip-tools must be installed in each of your project's
$ source /path/to/venv/bin/activate (venv) $ python -m pip install pip-tools
Note: all of the remaining example commands assume you've activated your project's virtual environment.
Example usage for
pip-compile command lets you compile a
requirements.txt file from
your dependencies, specified in either
Run it with
python -m piptools compile. If you use
multiple Python versions, you can also run
py -X.Y -m piptools compile on
pythonX.Y -m piptools compile on other systems.
pip-compile should be run from the same virtual environment as your
project so conditional dependencies that require a specific Python version,
or other environment markers, resolve relative to your project's
pip-compile finds an existing
requirements.txt file that
fulfils the dependencies then no changes will be made, even if updates are
available. To compile from scratch, first delete the existing
requirements.txt file, or see Updating requirements for alternative
pyproject.toml file is the
latest standard for configuring
packages and applications, and is recommended for new projects.
supports both installing your
project.dependencies as well as your
project.optional-dependencies. Thanks to the fact that this is an
official standard, you can use
pip-compile to pin the dependencies
in projects that use modern standards-adhering packaging tools like
Hatch or flit.
Suppose you have a Django application that is packaged using
Hatch, and you
want to pin it for production. You also want to pin your development tools
in a separate pin file. You declare
django as a dependency and create an
dev that includes
[build-system] requires = ["hatchling"] build-backend = "hatchling.build" [project] name = "my-cool-django-app" version = "42" dependencies = ["django"] [project.optional-dependencies] dev = ["pytest"]
You can produce your pin files as easily as:
$ pip-compile -o requirements.txt pyproject.toml # # This file is autogenerated by pip-compile with python 3.10 # To update, run: # # pip-compile --output-file=requirements.txt pyproject.toml # asgiref==3.5.2 # via django django==4.1 # via my-cool-django-app (pyproject.toml) sqlparse==0.4.2 # via django $ pip-compile --extra dev -o dev-requirements.txt pyproject.toml # # This file is autogenerated by pip-compile with python 3.10 # To update, run: # # pip-compile --extra=dev --output-file=dev-requirements.txt pyproject.toml # asgiref==3.5.2 # via django attrs==22.1.0 # via pytest django==4.1 # via my-cool-django-app (pyproject.toml) iniconfig==1.1.1 # via pytest packaging==21.3 # via pytest pluggy==1.0.0 # via pytest py==1.11.0 # via pytest pyparsing==3.0.9 # via packaging pytest==7.1.2 # via my-cool-django-app (pyproject.toml) sqlparse==0.4.2 # via django tomli==2.0.1 # via pytest
This is great for both pinning your applications, but also to keep the CI of your open-source Python package stable.
pip-compile has also full support for
setup.cfg-based projects that use
Just define your dependencies and extras as usual and run
pip-compile as above.
You can also use plain text files for your requirements (e.g. if you don't
want your application to be a package). To use a
requirements.in file to
declare the Django dependency:
# requirements.in django
$ pip-compile requirements.in # # This file is autogenerated by pip-compile # To update, run: # # pip-compile requirements.in # asgiref==3.2.3 # via django django==3.0.3 # via -r requirements.in pytz==2019.3 # via django sqlparse==0.3.0 # via django
And it will produce your
requirements.txt, with all the Django dependencies
(and all underlying dependencies) pinned.
pip-compile generates a
requirements.txt file using the latest versions
that fulfil the dependencies you specify in the supported files.
pip-compile finds an existing
requirements.txt file that fulfils the
dependencies then no changes will be made, even if updates are available.
pip-compile to update all packages in an existing
To update a specific package to the latest or a specific version use the
# only update the django package $ pip-compile --upgrade-package django # update both the django and requests packages $ pip-compile --upgrade-package django --upgrade-package requests # update the django package to the latest, and requests to v2.0.0 $ pip-compile --upgrade-package django --upgrade-package requests==2.0.0
You can combine
--upgrade-package in one command, to
provide constraints on the allowed upgrades. For example to upgrade all
packages whilst constraining requests to the latest version less than 3.0:
$ pip-compile --upgrade --upgrade-package 'requests<3.0'
If you would like to use Hash-Checking Mode available in
$ pip-compile --generate-hashes requirements.in # # This file is autogenerated by pip-compile # To update, run: # # pip-compile --generate-hashes requirements.in # asgiref==3.2.3 \ --hash=sha256:7e06d934a7718bf3975acbf87780ba678957b87c7adc056f13b6215d610695a0 \ --hash=sha256:ea448f92fc35a0ef4b1508f53a04c4670255a3f33d22a81c8fc9c872036adbe5 \ # via django django==3.0.3 \ --hash=sha256:2f1ba1db8648484dd5c238fb62504777b7ad090c81c5f1fd8d5eb5ec21b5f283 \ --hash=sha256:c91c91a7ad6ef67a874a4f76f58ba534f9208412692a840e1d125eb5c279cb0a \ # via -r requirements.in pytz==2019.3 \ --hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \ --hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be \ # via django sqlparse==0.3.0 \ --hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \ --hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873 \ # via django
To output the pinned requirements in a filename other than
--output-file. This might be useful for compiling
multiple files, for example with different constraints on django to test a
library with both versions using tox:
$ pip-compile --upgrade-package 'django<1.0' --output-file requirements-django0x.txt $ pip-compile --upgrade-package 'django<2.0' --output-file requirements-django1x.txt
Or to output to standard output, use
$ pip-compile --output-file=- > requirements.txt $ pip-compile - --output-file=- < requirements.in > requirements.txt
Forwarding options to
pip flags or arguments may be passed on with
--pip-args option, e.g.
$ pip-compile requirements.in --pip-args "--retries 10 --timeout 30"
You might be wrapping the
pip-compile command in another script. To avoid
confusing consumers of your custom script you can override the update command
generated at the top of requirements files by setting the
CUSTOM_COMPILE_COMMAND environment variable.
$ CUSTOM_COMPILE_COMMAND="./pipcompilewrapper" pip-compile requirements.in # # This file is autogenerated by pip-compile # To update, run: # # ./pipcompilewrapper # asgiref==3.2.3 # via django django==3.0.3 # via -r requirements.in pytz==2019.3 # via django sqlparse==0.3.0 # via django
Workflow for layered requirements
If you have different environments that you need to install different but compatible packages for, then you can create layered requirements files and use one layer to constrain the other.
For example, if you have a Django project where you want the newest
release in production and when developing you want to use the Django debug
toolbar, then you can create two
*.in files, one for each layer:
# requirements.in django<2.2
At the top of the development requirements
dev-requirements.in you use
requirements.txt to constrain the dev requirements to packages already
selected for production in
# dev-requirements.in -c requirements.txt django-debug-toolbar
requirements.txt as usual:
$ pip-compile # # This file is autogenerated by pip-compile # To update, run: # # pip-compile # django==2.1.15 # via -r requirements.in pytz==2019.3 # via django
Now compile the dev requirements and the
requirements.txt file is used as
$ pip-compile dev-requirements.in # # This file is autogenerated by pip-compile # To update, run: # # pip-compile dev-requirements.in # django-debug-toolbar==2.2 # via -r dev-requirements.in django==2.1.15 # via # -c requirements.txt # django-debug-toolbar pytz==2019.3 # via # -c requirements.txt # django sqlparse==0.3.0 # via django-debug-toolbar
As you can see above, even though a
2.2 release of Django is available, the
dev requirements only include a
2.1 version of Django because they were
constrained. Now both compiled requirements files can be installed safely in
the dev environment.
To install requirements in production stage use:
You can install requirements in development stage by:
$ pip-sync requirements.txt dev-requirements.txt
Version control integration
You might use
pip-compile as a hook for the pre-commit.
See pre-commit docs for instructions.
repos: - repo: https://github.com/jazzband/pip-tools rev: 6.12.3 hooks: - id: pip-compile
You might want to customize
pip-compile args by configuring
files, for example:
repos: - repo: https://github.com/jazzband/pip-tools rev: 6.12.3 hooks: - id: pip-compile files: ^requirements/production\.(in|txt)$ args: [--index-url=https://example.com, requirements/production.in]
If you have multiple requirement files make sure you create a hook for each file.
repos: - repo: https://github.com/jazzband/pip-tools rev: 6.12.3 hooks: - id: pip-compile name: pip-compile setup.py files: ^(setup\.py|requirements\.txt)$ - id: pip-compile name: pip-compile requirements-dev.in args: [requirements-dev.in] files: ^requirements-dev\.(in|txt)$ - id: pip-compile name: pip-compile requirements-lint.in args: [requirements-lint.in] files: ^requirements-lint\.(in|txt)$ - id: pip-compile name: pip-compile requirements.txt args: [requirements.txt] files: ^requirements\.(in|txt)$
Example usage for
Now that you have a
requirements.txt, you can use
pip-sync to update
your virtual environment to reflect exactly what's in there. This will
install/upgrade/uninstall everything necessary to match the
Run it with
python -m piptools sync. If you use multiple
Python versions, you can also run
py -X.Y -m piptools sync on Windows and
pythonX.Y -m piptools sync on other systems.
pip-sync must be installed into and run from the same virtual
environment as your project to identify which packages to install
pip-sync is meant to be used only with a
requirements.txt generated by
$ pip-sync Uninstalling flake8-2.4.1: Successfully uninstalled flake8-2.4.1 Collecting click==4.1 Downloading click-4.1-py2.py3-none-any.whl (62kB) 100% |................................| 65kB 1.8MB/s Found existing installation: click 4.0 Uninstalling click-4.0: Successfully uninstalled click-4.0 Successfully installed click-4.1
To sync multiple
*.txt dependency lists, just pass them in via command
line arguments, e.g.
$ pip-sync dev-requirements.txt requirements.txt
Passing in empty arguments would cause it to default to
pip install flags or arguments may be passed with
--pip-args option, e.g.
$ pip-sync requirements.txt --pip-args "--no-cache-dir --no-deps"
pip-sync will not upgrade or uninstall packaging tools like
pip-tools itself. Use
python -m pip install --upgrade
to upgrade those packages.
Should I commit
requirements.txt to source control?
Generally, yes. If you want a reproducible environment installation available from your source control,
then yes, you should commit both
requirements.txt to source control.
Note that if you are deploying on multiple Python environments (read the section below),
then you must commit a separate output file for each Python environment.
We suggest to use the
Cross-environment usage of
The dependencies of a package can change depending on the Python environment in which it is installed. Here, we define a Python environment as the combination of Operating System, Python version (3.7, 3.8, etc.), and Python implementation (CPython, PyPy, etc.). For an exact definition, refer to the possible combinations of PEP 508 environment markers.
As the resulting
requirements.txt can differ for each environment, users must
pip-compile on each Python environment separately to generate a
requirements.txt valid for each said environment. The same
be used as the source file for all environments, using PEP 508 environment markers as
needed, the same way it would be done for regular
pip cross-environment usage.
If the generated
requirements.txt remains exactly the same for all Python
environments, then it can be used across Python environments safely. But users
should be careful as any package update can introduce environment-dependent
dependencies, making any newly generated
requirements.txt environment-dependent too.
As a general rule, it's advised that users should still always execute
on each targeted Python environment to avoid issues.
Other useful tools
- pipdeptree to print the dependency tree of the installed packages.
- requirements.txt.vim for Vim.
- Python extension for VS Code for VS Code.
- pip-requirements.el for Emacs.
This section lists
pip-tools features that are currently deprecated.
- In future versions, the
--allow-unsafebehavior will be enabled by default. Use
--no-allow-unsafeto keep the old behavior. It is recommended to pass the
--allow-unsafenow to adapt to the upcoming change.
- Legacy resolver is deprecated and will be removed in future versions.
A Note on Resolvers
You can choose from either the legacy or the backtracking resolver. The backtracking resolver is recommended, and will become the default with the 7.0 release.
Use it now with the
--resolver=backtracking option to
The legacy resolver will occasionally fail to resolve dependencies. The backtracking resolver is more robust, but can take longer to run in general.
You can continue using the legacy resolver with
Versions and compatibility
The table below summarizes the latest
pip-tools versions with the required
pip and Python versions. Generally,
pip-tools supports the same Python
versions as the required
|4.5.*||8.1.3 - 20.0.2||2.7, 3.5 - 3.8|
|5.0.0 - 5.3.0||20.0 - 20.1.1||2.7, 3.5 - 3.8|
|5.4.0||20.1 - 20.3.*||2.7, 3.5 - 3.8|
|5.5.0||20.1 - 20.3.*||2.7, 3.5 - 3.9|
|6.0.0 - 6.3.1||20.3 - 21.2.*||3.6 - 3.9|
|6.4.0||21.2 - 21.3.*||3.6 - 3.10|
|6.5.0 - 6.10.0||21.2 - 22.3.*||3.7 - 3.11|
|6.11.0+||22.2+||3.7 - 3.11|