Update a handful of Python dependencies #6229

callahad · 2019-12-11T23:25:26Z

Part of #6173

Before: 71/113 (63%) of our Python dependencies are outdated.

After: 48/116 (41%) of our Python dependencies are outdated.

These should all be low-risk updates. I've added a detailed message to each commit including notes and links to relevant changelogs.

Updates colorama, a Pytest dependency for supporting ANSI escape sequences on Windows. pytest 3.1.3 pytest: simple powerful testing with Python |-- colorama * |-- py >=1.4.33 `-- setuptools * We do not use colorama directly in Kuma. There are no actual code changes between colorama 0.4.1 and 0.4.3; they were metadata-only updates: tartley/colorama#243 (comment)

Recategorized 11 dependencies that were mistakenly marked as constraints, but actually used directly in Kuma. Removed a duplicate entry for `urlwait`, which was leftover from our previous system for tracking dependencies.

- Updating pytz (2018.9 -> 2019.3) https://github.com/stub42/pytz/blob/release_2019.3/tz/NEWS No meaningful code changes, but keeps our timezone info current. Speaking of, I highly recommend reading the zoneinfo database some time, super interesting historical perspective. More relevantly, Brazil cancelled Daylight Savings Time since we last updated our copy of pytz.

- Updating amqp (2.4.2 -> 2.5.2) https://github.com/celery/py-amqp/blob/2.5.2/Changelog - Updating billiard (3.6.0.0 -> 3.6.1.0) https://github.com/celery/billiard/blob/3.6.1/CHANGES.txt - Updating kombu (4.5.0 -> 4.6.7) https://github.com/celery/kombu/blob/4.6.7/Changelog.rst We're already on the latest version of Celery itself. --- Celery 4.3.0's dependency tree: celery 4.3.0 Distributed Task Queue. |-- billiard >=3.6.0,<4.0 |-- kombu >=4.4.0,<5.0 | `-- amqp >=2.5.2,<2.6 | `-- vine >=1.1.3,<5.0.0a1 |-- pytz >0.0-dev `-- vine >=1.3.0 No red flags in the changelogs, and with the exception of pytz, none of Celery's dependencies appear elsewhere in our dependency tree.

- Updating boto3 (1.9.128 -> 1.10.37) https://github.com/boto/boto3/blob/1.10.37/CHANGELOG.rst - Updating botocore (1.12.130 -> 1.13.37) https://github.com/boto/botocore/blob/1.13.37/CHANGELOG.rst - Updating docutils (0.12 -> 0.15.2) http://docutils.sourceforge.net/RELEASE-NOTES.html - Updating python-dateutil (2.7.3 -> 2.8.0) https://github.com/dateutil/dateutil/blob/2.8.1/NEWS - Updating s3transfer (0.2.0 -> 0.2.1) https://github.com/boto/s3transfer/blob/0.2.1/CHANGELOG.rst - Updating six (1.11.0 -> 1.13.0) https://github.com/benjaminp/six/blob/1.13.0/CHANGES --- Boto3 1.9.128's dependency tree: boto3 1.9.128 The AWS SDK for Python |-- botocore >=1.12.128,<1.13.0 | |-- docutils >=0.10 | |-- jmespath >=0.7.1,<1.0.0 | |-- python-dateutil >=2.1,<3.0.0 | | `-- six >=1.5 | `-- urllib3 >=1.20,<1.25 |-- jmespath >=0.7.1,<1.0.0 `-- s3transfer >=0.2.0,<0.3.0 `-- botocore >=1.12.36,<2.0.0 |-- docutils >=0.10 |-- jmespath >=0.7.1,<1.0.0 |-- python-dateutil >=2.1,<3.0.0 | `-- six >=1.5 `-- urllib3 >=1.20,<1.25 The boto3 (1.9 -> 1.10) and botocore (1.12 -> 1.13) minor version numbers were incremented due to Boto removing its vendored copy of requests. Otherwise no significant changes to behavior. urllib3 was not updated from 1.24.3, despite 1.25.7 being available, because our pinned version of requests wants urllib3 (>=1.21.1,<1.25). Similarly, python-dateutil is now stuck at 2.8.0 because botocore wants python-dateutil (>=2.1,<2.8.1). Relaxing that upper bound is tracked at boto/botocore#1872. All of the changelogs look benign, and Boto is fairly self-contained. However, python-dateutil is also used by elasticsearch-dsl, and six is used in a few other dependencies.

- Updating certifi (2018.10.15 -> 2019.11.28) certifi/python-certifi@2019.03.09...2019.11.28 - Updating idna (2.7 -> 2.8) https://github.com/kjd/idna/blob/v2.8/HISTORY.rst - Updating requests (2.20.1 -> 2.22.0) https://github.com/psf/requests/blob/v2.22.0/HISTORY.md - Updating urllib3 (1.24.3 -> 1.25.7) https://github.com/urllib3/urllib3/blob/1.25.7/CHANGES.rst --- Requests 2.20.1's dependency tree: requests 2.20.1 Python HTTP for Humans. |-- certifi >=2017.4.17 |-- chardet >=3.0.2,<3.1.0 |-- idna >=2.5,<2.8 `-- urllib3 >=1.21.1,<1.25 Updating requests also allowed urllib3 to update; it was unpinned in the previous commit. All of the changes look reasonably minimal.

- Updating click (6.7 -> 7.0) https://github.com/pallets/click/blob/7.0/CHANGES.rst Only used by dennis, which we use for linting locale files

- Updating pyquery (1.4.0 -> 1.4.1) https://github.com/gawel/pyquery/blob/1.4.1/CHANGES.rst

- Updating cssselect (1.0.3 -> 1.1.0) https://github.com/scrapy/cssselect/blob/v1.1.0/CHANGES

- Updating lxml (4.2.5 -> 4.4.2) https://github.com/lxml/lxml/blob/lxml-4.4.2/CHANGES.txt As a bonus, lxml >= 4.4.1 provides wheels for CPython 3.8 on Linux, which should speed up our Docker image builds.

- Installing cachetools (3.1.1) https://github.com/tkem/cachetools - Updating google-api-python-client (1.5.1 -> 1.7.11) https://github.com/googleapis/google-api-python-client/blob/v1.7.11/CHANGELOG - Installing google-auth (1.8.2) https://github.com/googleapis/google-auth-library-python - Installing google-auth-httplib2 (0.0.3) https://github.com/GoogleCloudPlatform/google-auth-library-python-httplib2 - Updating httplib2 (0.9.2 -> 0.14.0) https://github.com/httplib2/httplib2/blob/v0.14.0/CHANGELOG - Updating oauth2client (3.0.0 -> 4.1.3) https://github.com/googleapis/oauth2client/blob/v4.1.3/CHANGELOG.md - Updating pyasn1 (0.1.9 -> 0.4.8) https://github.com/etingof/pyasn1/blob/v0.4.8/CHANGES.rst - Updating pyasn1-modules (0.0.8 -> 0.2.7) https://github.com/etingof/pyasn1-modules/blob/v0.2.8/CHANGES.txt - Updating rsa (3.4.2 -> 4.0) https://github.com/sybrenstuvel/python-rsa/blob/version-4.0/CHANGELOG.txt - Updating uritemplate (0.6 -> 3.0.0) https://github.com/python-hyper/uritemplate/blob/3.0.0/HISTORY.rst --- google-api-python-client 1.5.1's dependency tree: google-api-python-client 1.5.1 Google API Client Library for Python |-- httplib2 >=0.8,<1 |-- oauth2client * | |-- httplib2 >=0.9.1 | |-- pyasn1 >=0.1.7 | |-- pyasn1-modules >=0.0.5 | | `-- pyasn1 >=0.1.8 (circular dependency aborted here) | |-- rsa >=3.1.4 | | `-- pyasn1 >=0.1.3 (circular dependency aborted here) | `-- six >=1.6.1 |-- six >=1.6.1,<2 `-- uritemplate >=0.6,<1 google-api-python-client 1.7.11's dependency tree: google-api-python-client 1.7.11 Google API Client Library for Python |-- google-auth >=1.4.1 | |-- cachetools >=2.0.0,<3.2 | |-- pyasn1-modules >=0.2.1 | | `-- pyasn1 >=0.4.6,<0.5.0 | |-- rsa >=3.1.4,<4.1 | | `-- pyasn1 >=0.1.3 (circular dependency aborted here) | |-- setuptools >=40.3.0 | `-- six >=1.9.0 |-- google-auth-httplib2 >=0.0.3 | |-- google-auth * | | |-- cachetools >=2.0.0,<3.2 | | |-- pyasn1-modules >=0.2.1 | | | `-- pyasn1 >=0.4.6,<0.5.0 | | |-- rsa >=3.1.4,<4.1 | | | `-- pyasn1 >=0.1.3 (circular dependency aborted here) | | |-- setuptools >=40.3.0 | | `-- six >=1.9.0 | `-- httplib2 >=0.9.1 |-- httplib2 >=0.9.2,<1dev |-- six >=1.6.1,<2dev `-- uritemplate >=3.0.0,<4dev Note the removal of oauth2client and forking of httplib2. Both are explicitly discouraged. __oauth2client:__ > The long-term replacement for `oauth2client` is this library, > `google-auth`. This library addresses the major issues with oauthclient > > https://google-auth.readthedocs.io/en/latest/oauth2client-deprecation.html __httplib:__ > `httplib` has lots of problems such as lack of threadsafety and > insecure usage of TLS. Using it is highly discouraged. This library is > intended to help existing users of `oauth2client` migrate to > `google-auth`. > > https://github.com/GoogleCloudPlatform/google-auth-library-python-httplib2 We do not directly use pyasn1, pyasn1-modules, rsa, or uritemplate, so allowing those to freely update according to google-api-python-client's constraints.

This reverts commit c30a60e. Pyquery 1.4.1 is preserving leading whitespace in a way that breaks our tests. Revert for now and deal with in a separate PR.

codecov-io · 2019-12-12T00:33:25Z

Codecov Report

Merging #6229 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #6229   +/-   ##
=======================================
  Coverage   48.42%   48.42%           
=======================================
  Files         296      296           
  Lines       32738    32738           
  Branches     1729     1729           
=======================================
  Hits        15855    15855           
  Misses      16350    16350           
  Partials      533      533

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2f1060...83d6b43. Read the comment docs.

escattone

Thanks @callahad!

callahad added 12 commits December 11, 2019 23:19

Audit constraints

0a3ac6e

Recategorized 11 dependencies that were mistakenly marked as constraints, but actually used directly in Kuma. Removed a duplicate entry for `urlwait`, which was leftover from our previous system for tracking dependencies.

Unpin and update click

eec0f20

- Updating click (6.7 -> 7.0) https://github.com/pallets/click/blob/7.0/CHANGES.rst Only used by dennis, which we use for linting locale files

Update pyquery

c30a60e

- Updating pyquery (1.4.0 -> 1.4.1) https://github.com/gawel/pyquery/blob/1.4.1/CHANGES.rst

Update cssselect

5d3c12b

- Updating cssselect (1.0.3 -> 1.1.0) https://github.com/scrapy/cssselect/blob/v1.1.0/CHANGES

Update lxml

bad704a

- Updating lxml (4.2.5 -> 4.4.2) https://github.com/lxml/lxml/blob/lxml-4.4.2/CHANGES.txt As a bonus, lxml >= 4.4.1 provides wheels for CPython 3.8 on Linux, which should speed up our Docker image builds.

Revert "Update pyquery"

83d6b43

This reverts commit c30a60e. Pyquery 1.4.1 is preserving leading whitespace in a way that breaks our tests. Revert for now and deal with in a separate PR.

escattone approved these changes Dec 12, 2019

View reviewed changes

escattone merged commit c7829ae into mdn:master Dec 12, 2019

callahad deleted the update-deps branch January 31, 2020 08:49

tobinmori mentioned this pull request Feb 19, 2020

MultipleObjectsReturned "more than one EmailAddress" #6335

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update a handful of Python dependencies #6229

Update a handful of Python dependencies #6229

callahad commented Dec 11, 2019 •

edited

Loading

codecov-io commented Dec 12, 2019

escattone left a comment

Update a handful of Python dependencies #6229

Update a handful of Python dependencies #6229

Conversation

callahad commented Dec 11, 2019 • edited Loading

codecov-io commented Dec 12, 2019

Codecov Report

escattone left a comment

Choose a reason for hiding this comment

callahad commented Dec 11, 2019 •

edited

Loading