Skip to content
Permalink
Browse files

Pin all application requirements in requirements.txt

The list of top-level dependencies is moved to `reqirements-app.txt`
which is used to by `make compile-requirements` to generate the full
list of requirements in `requirements.txt`.

`requirements_for_test` and releated make step are renamed to
`requirements-dev` to reflect the fact that it should contain
any dependencies that aren't used by the app itself including
dependencies for tests, utility scripts and dev tools.

Rationale
---------

We've had a number of issues caused by unpinned dependencies (eg
#607). They can cause things to fail
at different stages depending on the root cause, but generally
highlight a few problems with our current approach:

* The latest version of unpinned packages are installed, so whenever
  there's a breaking change in a new package release we're forced to
  either update our code or pin the package alongside our top-level
  application dependencies.
* `pip install` is run at different times. Eg Travis CI will install
  dependencies on each test run but our image builds will only run
  it if `requirements.txt` have changed. This means that libraries
  we're running the tests with could be different from libraries that
  are used by the deployed application.
* Finding out which library release broke the build is hard because
  there's no easily available list of "known-good" versions. These
  could be extracted from the previous application container versions,
  but the process isn't documented.

The common solution to this is to ship a full list of pinned dependencies.
There are currently 3 ways to do this in Python:

1. Generating a `requirements.txt` file with `pip freeze`. This stores
   the full list of currently installed packages.
2. Using [pip-tools] to generate a full list of required packages from
   a list of top-level dependencies. pip-tools also provides utilities
   to upgrade package versions without modifying the files manually.
3. Defining dependencies in a [Pipfile] and using [pipenv] to manage
   pinned versions and the virtualenv.

[Pipfile] seems to be the planned replacement for requirements files,
however the tools appear to have some issues when applied to our repos.
For example, [pipenv] doesn't appear to be locking dependencies of the
VCS packages (eg utils and apiclient). It also doesn't detect non-semver
package versions correctly (eg `functools32==3.2.3.post2`). We could
fix the VCS dependencies issue by publishing our packages on PyPI, but
my feeling is the tools aren't ready for production just yet.

[pip-tools] appears to be more commonly used at the moment. It works by
examining setup.py files of packages listed in `requirements.in` file
and generating a full list of dependencies in `requirements.txt`.
The main issue with pip-tools when applied to our current applications
is that it's stricter than pip in resolving version conflicts. Some
of the packages we depend on are locking the libraries we use directly
in their own setup.py (eg `mandrill` requires `docopt==0.4`, while we
generally use `0.6.2`). While `pip` would install the latest package
version despite the conflict, pip-tools requires us to resolve the
conflict, which in this case means relying on a fork of mandrill client.
And pip's behaviour in this case isn't really a problem for us, since
docopt isn't used when mandrill is loaded as a module.

`pip freeze` is the simplest approach that doesn't require any
additional tools, however it also isn't very usably without additional
tooling. The main problems with using `pip freeze` directly is that it
will save whatever is installed in the current virtualenv including the
test packages and packages that have been installed manually and that it
loses VCS packages, only storing the names. This is solved by wrapping
`pip freeze` with a `make compile-requirements` step that:

1. Creates a temporary virtualenv.
2. Installs application requirements.
3. Runs `pip freeze` to get a full list of installed packages.
4. Modifies `pip freeze` output to maintain the list of application
   dependencies in the original format.

The benefits of this approach is that it keeps the existing process
of running `pip install -r requirements.txt` for application builds.
It also requires only one command to be run whenever application
dependencies change to regenerate `requirements.txt`.

The known downsides and unsolved problems:

* You need to remember to run `make compile-requirements` after
  changing `requirements-app.txt`
* Since the virtualenv is rebuilt from the application requirements
  it will still install newer versions of all unpinned packages. The
  difference is that the process is explicit and changes will be
  visible in the `requirements.txt` diff. This allows us to either
  accept the new package version or ignore it by discarding the change.
  This does mean that if there's a package we want to hold back we'll
  need to do this repeatedly after each `requirements-app.txt` change.
  The benefit of this approach is that it makes us aware of new package
  versions.

Another major point is that development requirements don't go through the
same process but are now loading the full list of application dependencies.
This in turn means that:

* `requirements-dev.txt` can't list any of the packages present in
  `requirements.txt` since pip will complain about a duplicate record.
  We can either rely on the record in `requirements.txt` in this case or
  move the dependency to `requirements-app.txt`
* While it's not possible for one of the dev requirements to override
  the application dependencies (pip's resolver prioritises the top-level
  dependency version) they do install additional packages that
  aren't present in the application deployment container. This is
  unchanged from the existing process but could potentially hide issues
  that are only discovered once the app has been deployed.

[pip-tools]: https://github.com/jazzband/pip-tools
[pipenv]: https://github.com/kennethreitz/pipenv
[Pipfile]: https://github.com/pypa/pipfile
  • Loading branch information...
allait committed Jul 12, 2017
1 parent 6bc7dab commit 95ac12206d26e6b219dd381dd63641c33467afbd
Showing with 94 additions and 10 deletions.
  1. +1 −1 .travis.yml
  2. +14 −3 Makefile
  3. +14 −1 README.md
  4. +15 −0 requirements-app.txt
  5. +2 −0 requirements_for_test.txt → requirements-dev.txt
  6. +47 −4 requirements.txt
  7. +1 −1 scripts/bootstrap.sh
@@ -8,7 +8,7 @@ dist: "trusty"
env:
- SQLALCHEMY_DATABASE_URI=postgresql://postgres:@localhost:5432/digitalmarketplace_test
install:
- make requirements_for_test
- make requirements-dev
before_script:
- psql -c 'create database digitalmarketplace_test;' -U postgres
script:
@@ -18,8 +18,19 @@ bootstrap: virtualenv
requirements: virtualenv requirements.txt
${VIRTUALENV_ROOT}/bin/pip install -r requirements.txt

requirements_for_test: virtualenv requirements_for_test.txt
${VIRTUALENV_ROOT}/bin/pip install -r requirements_for_test.txt
requirements-dev: virtualenv requirements-dev.txt
${VIRTUALENV_ROOT}/bin/pip install -r requirements-dev.txt

compile-requirements:
virtualenv venv-freeze
$$(pwd)/venv-freeze/bin/pip install -r requirements-app.txt
echo '# This file is autogenerated. Run `make compile-requirements`' > requirements.txt
echo '# to update it with any changes made in requirements-app.txt' >> requirements.txt
echo '' >> requirements.txt
cat requirements-app.txt >> requirements.txt
echo '' >> requirements.txt
$$(pwd)/venv-freeze/bin/pip freeze -r requirements-app.txt | sed -n '/The following requirements were added by pip freeze/,$$p' >> requirements.txt
rm -rf venv-freeze

test: test_pep8 test_migrations test_unit

@@ -43,4 +54,4 @@ docker-push:
docker push digitalmarketplace/api:${RELEASE_NAME}


.PHONY: virtualenv requirements requirements_for_test test_pep8 test_migrations test_unit test test_all run_migrations run_app run_all docker-build docker-push
.PHONY: virtualenv requirements requirements-dev compile-requirements test_pep8 test_migrations test_unit test test_all run_migrations run_app run_all docker-build docker-push
@@ -63,7 +63,7 @@ up to date by running upgrade.
Install new Python dependencies with pip
```make requirements_for_test```
```make requirements-dev```
### Run the tests
@@ -99,6 +99,19 @@ e.g.:
curl -i -H "Authorization: Bearer myToken" 127.0.0.1:5000/services
```
## Updating application dependencies
`requirements.txt` file is generated from the `requirements-app.txt` in order to pin
versions of all nested dependecies. If `requirements-app.txt` has been changed (or
we want to update the unpinned nested dependencies) `requirements.txt` should be
regenerated with
```
make compile-requirements
```
`requirements.txt` should be commited alongside `requirements-app.txt` changes.
## Using FeatureFlags
To use feature flags, check out the documentation in (the README of)
@@ -0,0 +1,15 @@
Flask==0.10.1
Flask-Bcrypt==0.7.1
Flask-Migrate==2.0.3
Flask-SQLAlchemy==2.1
psycopg2==2.5.4
SQLAlchemy==1.1.4
SQLAlchemy-Utils==0.30.5

git+https://github.com/alphagov/digitalmarketplace-utils.git@27.1.1#egg=digitalmarketplace-utils==27.1.1
git+https://github.com/alphagov/digitalmarketplace-apiclient.git@8.10.2#egg=digitalmarketplace-apiclient==8.10.2

# For schema validation
jsonschema==2.5.1
rfc3987==1.3.4
strict-rfc3339==0.5
@@ -1,4 +1,6 @@
-r requirements.txt

# For tests
pytest==2.8.2
pep8==1.5.7
requests-mock==0.6.0
@@ -1,7 +1,9 @@
# This file is autogenerated. Run `make compile-requirements`
# to update it with any changes made in requirements-app.txt

Flask==0.10.1
Flask-Bcrypt==0.7.1
Flask-Migrate==2.0.3
Flask-Script==2.0.5
Flask-SQLAlchemy==2.1
psycopg2==2.5.4
SQLAlchemy==1.1.4
@@ -15,7 +17,48 @@ jsonschema==2.5.1
rfc3987==1.3.4
strict-rfc3339==0.5

# For the import script
requests==2.9.1
docopt==0.6.2
## The following requirements were added by pip freeze:
alembic==0.9.3
backoff==1.0.7
bcrypt==3.1.3
boto3==1.4.4
botocore==1.5.80
cffi==1.10.0
contextlib2==0.4.0
cryptography==1.6
docopt==0.4.0
docutils==0.13.1
enum34==1.1.6
Flask-FeatureFlags==0.6
Flask-Script==2.0.5
Flask-WTF==0.12
functools32==3.2.3.post2
future==0.16.0
futures==3.1.1
idna==2.5
inflection==0.2.1
ipaddress==1.0.18
itsdangerous==0.24
Jinja2==2.9.6
jmespath==0.9.3
mailchimp3==2.0.11
Mako==1.0.6
mandrill==1.0.57
MarkupSafe==1.0
monotonic==0.3
notifications-python-client==4.1.0
pyasn1==0.2.3
pycparser==2.18
PyJWT==1.5.2
python-dateutil==2.6.1
python-editor==1.0.3
python-json-logger==0.1.4
pytz==2015.4
PyYAML==3.11
requests==2.7.0
s3transfer==0.1.10
six==1.10.0
unicodecsv==0.14.1
Werkzeug==0.12.2
workdays==1.4
WTForms==2.1
@@ -22,4 +22,4 @@ createdb digitalmarketplace_test

# Install Python development dependencies
# Upgrade databases
make requirements_for_test run_migrations
make requirements-dev run_migrations

0 comments on commit 95ac122

Please sign in to comment.
You can’t perform that action at this time.