New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use ZipFile.open instead of ZipFile.read #5848

Merged
merged 4 commits into from Oct 19, 2018

Conversation

Projects
None yet
4 participants
@waveform80
Copy link
Contributor

waveform80 commented Oct 4, 2018

To avoid huge memory usage in unusual situations (e.g. the TensorFlow wheel on a Raspberry Pi), use ZipFile.open and shutil.copyfileobj instead of ZipFile.read, which loads all the decompressed data of the file (an arbitrarily large amount) into a byte-string. On the Pi, this can lead to out-of-memory situations when there's no swap (the default configuration of Raspbian), and the wheel contains huge files.

Use ZipFile.open instead of ZipFile.read
To avoid huge memory usage in unusual situations (e.g. a TensorFlow
wheel on a Raspberry Pi), use ZipFile.open and shutil.copyfileobj
instead of reading all the decompressed data into a byte-string.
@waveform80

This comment has been minimized.

Copy link
Contributor Author

waveform80 commented Oct 4, 2018

I'm assuming for now that this would be marked trivial - do let me know if you want me to add a news file.

@cjerdonek

This comment has been minimized.

Copy link
Member

cjerdonek commented Oct 12, 2018

Thanks! I think it would be worth a news file IMO as some users will see a concrete benefit.

@waveform80

This comment has been minimized.

Copy link
Contributor Author

waveform80 commented Oct 12, 2018

Thanks! I think it would be worth a news file IMO as some users will see a concrete benefit.

Okay, will do.

waveform80 added some commits Oct 12, 2018

Address comments in PR
Add news file and comment to zip.open line
@cjerdonek

This comment has been minimized.

Copy link
Member

cjerdonek commented Oct 17, 2018

This PR looks okay to me now. Thanks, @waveform80!

However, in the course of reviewing the PR, I noticed that if you comment out the lines changed by this PR, the unzip_file() file test still passes. Thus, I made a separate PR to test the file contents: #5891 (I would feel bad about asking you, @waveform80, to make yet another change since you've been so patient!) So for completeness, I'd like to merge that PR before merging this one.

A couple other random things I noticed (but not for this PR as it's out of scope):

  1. The Python docs say it might be better to pass the ZipInfo object to open() rather than the name.
  2. Maybe it would be cleaner to use extract() at a future date rather than shutil.copyfileobj() (which extract() does under the hood anyways).
@waveform80

This comment has been minimized.

Copy link
Contributor Author

waveform80 commented Oct 18, 2018

The Python docs say it might be better to pass the ZipInfo object to open() rather than the name.

Yup - I considered adding that change to this PR too, but I'm always wary of unintended consequences, so I wound up keeping things as straight-forward as I could.

As I understand it, such a change should only affect pathological zip-files containing multiple members with the same path and name, which I think one could reasonably assume would never happen with a normally constructed wheel (I hope?).

The question then becomes what happens in each case and which behaviour is desirable; my assumption (unconfirmed) is that when using ZipInfo objects (and assuming the code iterates over the members in order), the last member with the duplicate name winds up being the file on disk (assuming it's opened with mode "w" which'd truncate any pre-existing file(s); if one used "w+" a mish-mash of content could result). Whereas, I'd assume when using filenames (as in the current code), the first member with the duplicate name winds up being the file on disk (because the other duplicate(s) are never reached). Given we'd presumably be talking about a maliciously constructed wheel anyway, I'm not sure there's a qualitative difference between the two options: pip has no way of knowing which member is "right" (if there's even such a thing).

Still, it might be worth considering the change anyway as it's probably a little more efficient (fewer filename lookups) and cleaner code.

Maybe it would be cleaner to use extract() at a future date rather than shutil.copyfileobj() (which extract() does under the hood anyways).

That was another change I considered but decided was probably out of scope for this PR. It's definitely worth thinking about though, given pip is sometimes run as root and I don't currently see much defence against a maliciously constructed wheel that could be used to overwrite stuff in, say, /bin. However, I'm also not sure if any of the filename manipulation that extract does (excising parent .. components, and substituting illegal characters in Windows) break existing assumptions in pip (I'd hope not :).

Anyway, there's some random thoughts on future changes for this; I'm happy to bash together a separate PR for them if you'd like.

@cjerdonek

This comment has been minimized.

Copy link
Member

cjerdonek commented Oct 19, 2018

I'm happy to bash together a separate PR for them if you'd like.

Certainly. You could create one or more PR's for each of those things if you'd like. We'll have to decide whether the risk outweighs the "reward" though.

A few other things that come to mind:

  1. Would you be able to review my PR #5891 and let me know if it looks okay to you? That will assist in landing this.

  2. Something else I could use help with (assuming you can reproduce the issue) is tracking down why checking the contents of "symlink.txt" didn't work in PR #5891 when I ran the tests locally on Mac OS X. The results were different for test_unpack_tgz() and test_unpack_zip(). Supplementing the test with that check is something else that could be done in a later PR.

  3. Given that you put some thought into duplicate names, you might be interested in taking a look at PR #5794. That PR adds test cases for has_leading_dir(), but it's not clear to me if the test expectations are what we want for the edge case of duplicate names..

@cjerdonek

This comment has been minimized.

Copy link
Member

cjerdonek commented Oct 19, 2018

Closing / reopening to trigger a new test run against latest master.

@cjerdonek cjerdonek closed this Oct 19, 2018

@cjerdonek cjerdonek reopened this Oct 19, 2018

@pradyunsg

This comment has been minimized.

Copy link
Member

pradyunsg commented Oct 19, 2018

@cjerdonek The test run on Travis does not check against the merged branch IIRC. AppVeyor does.

@cjerdonek

This comment has been minimized.

Copy link
Member

cjerdonek commented Oct 19, 2018

@pradyunsg I checked Travis beforehand, and it appears to do so. Like in this case, it's testing:
ecc070a

@cjerdonek cjerdonek merged commit 62c27de into pypa:master Oct 19, 2018

3 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
news-file/pr News files updated and/or change is trivial.
Details
@cjerdonek

This comment has been minimized.

Copy link
Member

cjerdonek commented Oct 19, 2018

Thanks for your work on this, @waveform80!

@waveform80 waveform80 deleted the waveform80:huge-wheels-small-platforms branch Oct 19, 2018

@waveform80

This comment has been minimized.

Copy link
Contributor Author

waveform80 commented Oct 19, 2018

  1. Would you be able to review my PR #5891 and let me know if it looks okay to you? That will assist in landing this.

Ah, beat me to it!

  1. Something else I could use help with (assuming you can reproduce the issue) is tracking down why checking the contents of "symlink.txt" didn't work in PR #5891 when I ran the tests locally on Mac OS X. The results were different for test_unpack_tgz() and test_unpack_zip(). Supplementing the test with that check is something else that could be done in a later PR.

Just had a look at this; I don't have a Mac but I can take a fair guess what's going on here: someone's expected symlinks in zips to "just work".

The zip spec itself has no clue about symlinks (unsurprising given its DOS origins). There's an "external attributes" field in the archive member meta-data in zips which several tools (infozip, and Python's ZipFile class) use to store the file mode (and thus the fact that a member might actually be a symlink). However, in order to re-create a symlink an extractor would still need to check the external attributes represent a symlink and call symlink() instead of open(). Neither pip's unzip_file nor Python's ZipFile.extractall do this, so they'll just extract symlinks as regular files containing the target (the infozip tools on Linux do, though).

If pip's expecting symlinks in a zip to work (which seems to be the case, given that test case, and the contents of test_zip.zip) then unzip_file definitely needs some work. Rather than clutter this ticket any further I'll throw another PR together and we can continue discussion there.

  1. Given that you put some thought into duplicate names, you might be interested in taking a look at PR #5794. That PR adds test cases for has_leading_dir(), but it's not clear to me if the test expectations are what we want for the edge case of duplicate names..

I'll take a look

bors bot added a commit to mozilla/normandy that referenced this pull request Feb 7, 2019

Merge #1723
1723: Scheduled weekly dependency update for week 05 r=mythmon a=pyup-bot






### Update [atomicwrites](https://pypi.org/project/atomicwrites) from **1.2.1** to **1.3.0**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/atomicwrites
  - Repo: https://github.com/untitaker/python-atomicwrites
</details>





### Update [botocore](https://pypi.org/project/botocore) from **1.12.82** to **1.12.86**.


<details>
  <summary>Changelog</summary>
  
  
   ### 1.12.86
   ```
   =======

* api-change:``devicefarm``: Update devicefarm client to latest version
* api-change:``codecommit``: Update codecommit client to latest version
* api-change:``medialive``: Update medialive client to latest version
* api-change:``mediaconnect``: Update mediaconnect client to latest version
   ```
   
  
  
   ### 1.12.85
   ```
   =======

* api-change:``logs``: Update logs client to latest version
* api-change:``elbv2``: Update elbv2 client to latest version
* api-change:``rds``: Update rds client to latest version
* api-change:``codebuild``: Update codebuild client to latest version
* api-change:``sms-voice``: Update sms-voice client to latest version
* api-change:``ecr``: Update ecr client to latest version
   ```
   
  
  
   ### 1.12.84
   ```
   =======

* api-change:``worklink``: Update worklink client to latest version
* api-change:``apigatewaymanagementapi``: Update apigatewaymanagementapi client to latest version
* api-change:``acm-pca``: Update acm-pca client to latest version
   ```
   
  
  
   ### 1.12.83
   ```
   =======

* api-change:``appstream``: Update appstream client to latest version
* api-change:``discovery``: Update discovery client to latest version
* api-change:``dms``: Update dms client to latest version
* api-change:``fms``: Update fms client to latest version
* api-change:``ssm``: Update ssm client to latest version
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/botocore
  - Changelog: https://pyup.io/changelogs/botocore/
  - Repo: https://github.com/boto/botocore
</details>





### Update [Faker](https://pypi.org/project/Faker) from **1.0.1** to **1.0.2**.


<details>
  <summary>Changelog</summary>
  
  
   ### 1.0.2
   ```
   --------------------------------------------------------------------------------------

* Fix state abbreviations for ``id_ID`` to be 2-letters. Thanks dt-ap.
* Fix format for ``city_with_postcode`` on ``de_DE`` locale. Thanks TZanke.
* Update ``person`` providers for ``zh_CN``. Thanks TimeFinger.
* Implement ``zipcode_in_state`` and aliases in ``en_US`` locale for generating
  a zipcode for a specified state. Thanks mattyg.
* Group first names by gender on ``zh_CN`` provider. Thanks TimeFinger.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/faker
  - Changelog: https://pyup.io/changelogs/faker/
  - Repo: https://github.com/joke2k/faker
</details>





### Update [pycodestyle](https://pypi.org/project/pycodestyle) from **2.4.0** to **2.5.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 2.5.0
   ```
   ------------------

New checks:

* E117: Over-indented code blocks
* W505: Maximum doc-string length only when configured with --max-doc-length

Changes:

* Remove support for EOL Python 2.6 and 3.3. PR 720.
* Add E117 error for over-indented code blocks.
* Allow W605 to be silenced by ` noqa` and fix the position reported by W605
* Allow users to omit blank lines around one-liner definitions of classes and
  functions
* Include the function return annotation (``-&gt;``) as requiring surrounding
  whitespace only on Python 3
* Verify that only names can follow ``await``. Previously we allowed numbers
  and strings.
* Add support for Python 3.7
* Fix detection of annotated argument defaults for E252
* Cprrect the position reported by W504
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pycodestyle
  - Changelog: https://pyup.io/changelogs/pycodestyle/
  - Docs: https://pycodestyle.readthedocs.io/
</details>





### Update [pyflakes](https://pypi.org/project/pyflakes) from **2.0.0** to **2.1.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 2.1.0
   ```
   - Allow intentional assignment to variables named ``_``
- Recognize ``__module__`` as a valid name in class scope
- ``pyflakes.checker.Checker`` supports checking of partial ``ast`` trees
- Detect assign-before-use for local variables which shadow builtin names
- Detect invalid ``print`` syntax using ``&gt;&gt;`` operator
- Treat ``async for`` the same as a ``for`` loop for introducing variables
- Add detection for list concatenation in ``__all__``
- Exempt ``typing.overload`` from duplicate function declaration
- Importing a submodule of an ``as``-aliased ``import``-import is marked as
  used
- Report undefined names from ``__all__`` as possibly coming from a ``*``
  import
- Add support for changes in Python 3.8-dev
- Add support for PEP 563 (``from __future__ import annotations``)
- Include Python version and platform information in ``pyflakes --version``
- Recognize ``__annotations__`` as a valid magic global in Python 3.6+
- Mark names used in PEP 484 `` type: ...`` comments as used
- Add check for use of ``is`` operator with ``str``, ``bytes``, and ``int``
  literals
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pyflakes
  - Changelog: https://pyup.io/changelogs/pyflakes/
  - Repo: https://github.com/PyCQA/pyflakes
</details>





### Update [setuptools](https://pypi.org/project/setuptools) from **40.6.3** to **40.7.3**.


<details>
  <summary>Changelog</summary>
  
  
   ### 40.7.3
   ```
   -------

* 1670: In package_index, revert to using a copy of splituser from Python 3.8. Attempts to use ``urllib.parse.urlparse`` led to problems as reported in 1663 and 1668. This change serves as an alternative to 1499 and fixes 1668.
   ```
   
  
  
   ### 40.7.2
   ```
   -------

* 1666: Restore port in URL handling in package_index.
   ```
   
  
  
   ### 40.7.1
   ```
   -------

* 1660: On Python 2, when reading config files, downcast options from text to bytes to satisfy distutils expectations.
   ```
   
  
  
   ### 40.7.0
   ```
   -------

* 1551: File inputs for the `license` field in `setup.cfg` files now explicitly raise an error.
* 1180: Add support for non-ASCII in setup.cfg (1062). Add support for native strings on some parameters (1136).
* 1499: ``setuptools.package_index`` no longer relies on the deprecated ``urllib.parse.splituser`` per Python 27485.
* 1544: Added tests for PackageIndex.download (for git URLs).
* 1625: In PEP 517 build_meta builder, ensure that sdists are built as gztar per the spec.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/setuptools
  - Changelog: https://pyup.io/changelogs/setuptools/
  - Repo: https://github.com/pypa/setuptools
</details>





### Update [cachetools](https://pypi.org/project/cachetools) from **3.0.0** to **3.1.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 3.1.0
   ```
   -------------------

- Fix Python 3.8 compatibility issue.

- Use ``time.monotonic`` as default timer if available.

- Improve documentation regarding thread safety.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/cachetools
  - Changelog: https://pyup.io/changelogs/cachetools/
  - Repo: https://github.com/tkem/cachetools
</details>





### Update [boto3](https://pypi.org/project/boto3) from **1.9.82** to **1.9.86**.


<details>
  <summary>Changelog</summary>
  
  
   ### 1.9.86
   ```
   ======

* api-change:``devicefarm``: [``botocore``] Update devicefarm client to latest version
* api-change:``codecommit``: [``botocore``] Update codecommit client to latest version
* api-change:``medialive``: [``botocore``] Update medialive client to latest version
* api-change:``mediaconnect``: [``botocore``] Update mediaconnect client to latest version
   ```
   
  
  
   ### 1.9.85
   ```
   ======

* api-change:``logs``: [``botocore``] Update logs client to latest version
* api-change:``elbv2``: [``botocore``] Update elbv2 client to latest version
* api-change:``rds``: [``botocore``] Update rds client to latest version
* api-change:``codebuild``: [``botocore``] Update codebuild client to latest version
* api-change:``sms-voice``: [``botocore``] Update sms-voice client to latest version
* api-change:``ecr``: [``botocore``] Update ecr client to latest version
   ```
   
  
  
   ### 1.9.84
   ```
   ======

* api-change:``worklink``: [``botocore``] Update worklink client to latest version
* api-change:``apigatewaymanagementapi``: [``botocore``] Update apigatewaymanagementapi client to latest version
* api-change:``acm-pca``: [``botocore``] Update acm-pca client to latest version
   ```
   
  
  
   ### 1.9.83
   ```
   ======

* api-change:``appstream``: [``botocore``] Update appstream client to latest version
* api-change:``discovery``: [``botocore``] Update discovery client to latest version
* api-change:``dms``: [``botocore``] Update dms client to latest version
* api-change:``fms``: [``botocore``] Update fms client to latest version
* api-change:``ssm``: [``botocore``] Update ssm client to latest version
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/boto3
  - Changelog: https://pyup.io/changelogs/boto3/
  - Repo: https://github.com/boto/boto3
</details>





### Update [flake8](https://pypi.org/project/flake8) from **3.6.0** to **3.7.5**.


<details>
  <summary>Changelog</summary>
  
  
   ### 3.7.5
   ```
   -------------------

You can view the `3.7.5 milestone`_ on GitLab for more details.

Bugs Fixed
~~~~~~~~~~

- Fix reporting of pyflakes &quot;referenced before assignment&quot; error (See also
  `GitLab!301`_, `GitLab503`_)


.. all links
.. _3.7.5 milestone:
    https://gitlab.com/pycqa/flake8/milestones/28

.. issue links
.. _GitLab503:
    https://gitlab.com/pycqa/flake8/issues/503

.. merge request links
.. _GitLab!301:
    https://gitlab.com/pycqa/flake8/merge_requests/301
   ```
   
  
  
   ### 3.7.4
   ```
   -------------------

You can view the `3.7.4 milestone`_ on GitLab for more details.

Bugs Fixed
~~~~~~~~~~

- Fix performance regression with lots of ``per-file-ignores`` and errors
  (See also `GitLab!299`_, `GitLab501`_)


.. all links
.. _3.7.4 milestone:
    https://gitlab.com/pycqa/flake8/milestones/27

.. issue links
.. _GitLab501:
    https://gitlab.com/pycqa/flake8/issues/501

.. merge request links
.. _GitLab!299:
    https://gitlab.com/pycqa/flake8/merge_requests/299
   ```
   
  
  
   ### 3.7.3
   ```
   -------------------

You can view the `3.7.3 milestone`_ on GitLab for more details.

Bugs Fixed
~~~~~~~~~~

- Fix imports of ``typing`` in python 3.5.0 / 3.5.1 (See also `GitLab!294`_,
  `GitLab498`_)

- Fix ``flake8 --statistics`` (See also `GitLab!295`_, `GitLab499`_)

- Gracefully ignore ``flake8-per-file-ignores`` plugin if installed (See also
  `GitLab!297`_, `GitLab495`_)

- Improve error message for malformed ``per-file-ignores`` (See also
  `GitLab!298`_, `GitLab489`_)


.. all links
.. _3.7.3 milestone:
    https://gitlab.com/pycqa/flake8/milestones/26

.. issue links
.. _GitLab489:
    https://gitlab.com/pycqa/flake8/issues/489
.. _GitLab495:
    https://gitlab.com/pycqa/flake8/issues/495
.. _GitLab498:
    https://gitlab.com/pycqa/flake8/issues/498
.. _GitLab499:
    https://gitlab.com/pycqa/flake8/issues/499

.. merge request links
.. _GitLab!294:
    https://gitlab.com/pycqa/flake8/merge_requests/294
.. _GitLab!295:
    https://gitlab.com/pycqa/flake8/merge_requests/295
.. _GitLab!297:
    https://gitlab.com/pycqa/flake8/merge_requests/297
.. _GitLab!298:
    https://gitlab.com/pycqa/flake8/merge_requests/298
   ```
   
  
  
   ### 3.7.2
   ```
   -------------------

You can view the `3.7.2 milestone`_ on GitLab for more details.

Bugs Fixed
~~~~~~~~~~

- Fix broken ``flake8 --diff`` (regressed in 3.7.0) (See also `GitLab!292`_,
  `GitLab490`_)

- Fix typo in plugin exception reporting (See also `GitLab!275`_,
  `GitLab491`_)

- Fix ``AttributeError`` while attempting to use the legacy api (regressed in
  3.7.0) (See also `GitLab!293`_, `GitLab497`_)

.. all links
.. _3.7.2 milestone:
    https://gitlab.com/pycqa/flake8/milestones/25

.. issue links
.. _GitLab490:
    https://gitlab.com/pycqa/flake8/issues/490
.. _GitLab491:
    https://gitlab.com/pycqa/flake8/issues/491
.. _GitLab497:
    https://gitlab.com/pycqa/flake8/issues/497

.. merge request links
.. _GitLab!292:
    https://gitlab.com/pycqa/flake8/merge_requests/292
.. _GitLab!275:
    https://gitlab.com/pycqa/flake8/merge_requests/275
.. _GitLab!293:
    https://gitlab.com/pycqa/flake8/merge_requests/293
   ```
   
  
  
   ### 3.7.1
   ```
   -------------------

You can view the `3.7.1 milestone`_ on GitLab for more details.

Bugs Fixed
~~~~~~~~~~

- Fix capitalized filenames in ``per-file-ignores`` setting (See also
  `GitLab!290`_, `GitLab488`_)

.. all links
.. _3.7.1 milestone:
    https://gitlab.com/pycqa/flake8/milestones/24

.. issue links
.. _GitLab488:
    https://gitlab.com/pycqa/flake8/issues/488

.. merge request links
.. _GitLab!290:
    https://gitlab.com/pycqa/flake8/merge_requests/290
   ```
   
  
  
   ### 3.7.0
   ```
   -------------------

You can view the `3.7.0 milestone`_ on GitLab for more details.

New Dependency Information
~~~~~~~~~~~~~~~~~~~~~~~~~~

- Add dependency on ``entrypoints`` &gt;= 0.3, &lt; 0.4 (See also `GitLab!264`_,
  `GitLab!288`_)

- Pyflakes has been updated to &gt;= 2.1.0, &lt; 2.2.0 (See also `GitLab!283`_,
  `GitLab!285`_)

- pycodestyle has been updated to &gt;= 2.5.0, &lt; 2.6.0 (See also `GitLab!287`_)

Features
~~~~~~~~

- Add support for ``per-file-ignores`` (See also `GitLab!259`_, `GitLab156`_,
  `GitLab!281`_, `GitLab471`_)

- Enable use of ``float`` and ``complex`` option types (See also `GitLab!261`_,
  `GitLab452`_)

- Improve startup performance by switching from ``pkg_resources`` to
  ``entrypoints`` (See also `GitLab!264`_)

- Add metadata for use through the `pre-commit`_ git hooks framework (See also
  `GitLab!268`_, `GitLab!284`_)

- Allow physical line checks to return more than one result (See also
  `GitLab!269`_)

- Allow `` noqa:X123`` comments without space between the colon and codes
  list (See also `GitLab!273`_, `GitLab470`_)

- Remove broken and unused ``flake8.listen`` plugin type (See also
  `GitLab!274`_, `GitLab480`_)

.. all links
.. _3.7.0 milestone:
    https://gitlab.com/pycqa/flake8/milestones/23
.. _pre-commit:
    https://pre-commit.com/

.. issue links
.. _GitLab156:
    https://gitlab.com/pycqa/flake8/issues/156
.. _GitLab452:
    https://gitlab.com/pycqa/flake8/issues/452
.. _GitLab470:
    https://gitlab.com/pycqa/flake8/issues/470
.. _GitLab471:
    https://gitlab.com/pycqa/flake8/issues/471
.. _GitLab480:
    https://gitlab.com/pycqa/flake8/issues/480

.. merge request links
.. _GitLab!259:
    https://gitlab.com/pycqa/flake8/merge_requests/259
.. _GitLab!261:
    https://gitlab.com/pycqa/flake8/merge_requests/261
.. _GitLab!264:
    https://gitlab.com/pycqa/flake8/merge_requests/264
.. _GitLab!268:
    https://gitlab.com/pycqa/flake8/merge_requests/268
.. _GitLab!269:
    https://gitlab.com/pycqa/flake8/merge_requests/269
.. _GitLab!273:
    https://gitlab.com/pycqa/flake8/merge_requests/273
.. _GitLab!274:
    https://gitlab.com/pycqa/flake8/merge_requests/274
.. _GitLab!281:
    https://gitlab.com/pycqa/flake8/merge_requests/281
.. _GitLab!283:
    https://gitlab.com/pycqa/flake8/merge_requests/283
.. _GitLab!284:
    https://gitlab.com/pycqa/flake8/merge_requests/284
.. _GitLab!285:
    https://gitlab.com/pycqa/flake8/merge_requests/285
.. _GitLab!287:
    https://gitlab.com/pycqa/flake8/merge_requests/287
.. _GitLab!288:
    https://gitlab.com/pycqa/flake8/merge_requests/288
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/flake8
  - Changelog: https://pyup.io/changelogs/flake8/
  - Repo: https://gitlab.com/pycqa/flake8
</details>





### Update [newrelic](https://pypi.org/project/newrelic) from **4.10.0.112** to **4.12.0.113**.


<details>
  <summary>Changelog</summary>
  
  
   ### 4.12.0.113
   ```
   This release of the Python agent extends support of Amazon&#39;s boto3 library and includes bug fixes.

The agent can be installed using easy_install/pip/distribute via the Python Package Index or can be downloaded directly from the New Relic download site.

Features


AWS operation and request ID will now be reported in transaction traces and
spans when using boto3 and botocore.

The agent will now report aws.requestId and aws.operation for all calls
to AWS made using botocore and boto3.
DynamoDB calls are now reported under the Databases tab.

The agent will now record DynamoDB query performance in the Databases tab in
APM in addition to table name for the following calls:


put_item
get_item
update_item
delete_item
create_table
delete_table
query
scan

Certain SQS calls will now report additional data for spans and transaction
traces.

The agent will now record the queue name in spans and transaction traces for
the following SQS calls:


send_message
send_message_batch
receive_message

SNS publish will now report additional data for spans and transaction traces.

The SNS topic, target, or the string literal PhoneNumber will be reported to
New Relic inside of spans and transaction traces.
The full URL path will now be recorded on span events and transaction traces
when using boto3 or botocore.

The agent will now record the full URL path for API calls made to AWS through
the boto3 / botocore libraries. The path will be available through span
events and transaction traces.


Bug Fixes


Using newrelic-admin to start a GunicornWebWorker with an application factory
resulted in an application crash.

The agent would fail to start if using the newrelic-admin command to start an
aiohttp application factory with GunicornWebWorker. This issue has now been
fixed.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/newrelic
  - Changelog: https://pyup.io/changelogs/newrelic/
  - Homepage: http://newrelic.com/docs/python/new-relic-for-python
</details>





### Update [psycopg2](https://pypi.org/project/psycopg2) from **2.7.6.1** to **2.7.7**.


<details>
  <summary>Changelog</summary>
  
  
   ### 2.7.7
   ```
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^

- Cleanup of the cursor results assignment code, which might have solved
  double free and inconsistencies in concurrent usage (🎟`346, 384`).
- Wheel package compiled against OpenSSL 1.0.2q.
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/psycopg2
  - Changelog: https://pyup.io/changelogs/psycopg2/
  - Homepage: http://initd.org/psycopg/
</details>





### Update [pyasn1-modules](https://pypi.org/project/pyasn1-modules) from **0.2.3** to **0.2.4**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pyasn1-modules
  - Changelog: https://pyup.io/changelogs/pyasn1-modules/
  - Repo: https://github.com/etingof/pyasn1-modules
</details>





### Update [pytest-django](https://pypi.org/project/pytest-django) from **3.4.5** to **3.4.7**.


<details>
  <summary>Changelog</summary>
  
  
   ### 3.4.7
   ```
   ------------------

Bugfixes
^^^^^^^^

* Fix disabling/handling of unittest methods with pytest 4.2+ (700)
   ```
   
  
  
   ### 3.4.6
   ```
   ------------------

Bugfixes
^^^^^^^^

* django_find_project: add cwd as fallback always (690)

Misc
^^^^

* Enable tests for Django 2.2 and add classifier (693)
* Disallow pytest 4.2.0 in ``install_requires`` (697)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest-django
  - Changelog: https://pyup.io/changelogs/pytest-django/
  - Docs: https://pytest-django.readthedocs.io/
</details>





### Update [pytest](https://pypi.org/project/pytest) from **4.1.1** to **4.2.0**.


<details>
  <summary>Changelog</summary>
  
  
   ### 4.2.0
   ```
   =========================

Features
--------

- `3094 &lt;https://github.com/pytest-dev/pytest/issues/3094&gt;`_: `Class xunit-style &lt;https://docs.pytest.org/en/latest/xunit_setup.html&gt;`__ functions and methods
  now obey the scope of *autouse* fixtures.

  This fixes a number of surprising issues like ``setup_method`` being called before session-scoped
  autouse fixtures (see `517 &lt;https://github.com/pytest-dev/pytest/issues/517&gt;`__ for an example).


- `4627 &lt;https://github.com/pytest-dev/pytest/issues/4627&gt;`_: Display a message at the end of the test session when running under Python 2.7 and 3.4 that pytest 5.0 will no longer
  support those Python versions.


- `4660 &lt;https://github.com/pytest-dev/pytest/issues/4660&gt;`_: The number of *selected* tests now are also displayed when the ``-k`` or ``-m`` flags are used.


- `4688 &lt;https://github.com/pytest-dev/pytest/issues/4688&gt;`_: ``pytest_report_teststatus`` hook now can also receive a ``config`` parameter.


- `4691 &lt;https://github.com/pytest-dev/pytest/issues/4691&gt;`_: ``pytest_terminal_summary`` hook now can also receive a ``config`` parameter.



Bug Fixes
---------

- `3547 &lt;https://github.com/pytest-dev/pytest/issues/3547&gt;`_: ``--junitxml`` can emit XML compatible with Jenkins xUnit.
  ``junit_family`` INI option accepts ``legacy|xunit1``, which produces old style output, and ``xunit2`` that conforms more strictly to https://github.com/jenkinsci/xunit-plugin/blob/xunit-2.3.2/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd


- `4280 &lt;https://github.com/pytest-dev/pytest/issues/4280&gt;`_: Improve quitting from pdb, especially with ``--trace``.

  Using ``q[quit]`` after ``pdb.set_trace()`` will quit pytest also.


- `4402 &lt;https://github.com/pytest-dev/pytest/issues/4402&gt;`_: Warning summary now groups warnings by message instead of by test id.

  This makes the output more compact and better conveys the general idea of how much code is
  actually generating warnings, instead of how many tests call that code.


- `4536 &lt;https://github.com/pytest-dev/pytest/issues/4536&gt;`_: ``monkeypatch.delattr`` handles class descriptors like ``staticmethod``/``classmethod``.


- `4649 &lt;https://github.com/pytest-dev/pytest/issues/4649&gt;`_: Restore marks being considered keywords for keyword expressions.


- `4653 &lt;https://github.com/pytest-dev/pytest/issues/4653&gt;`_: ``tmp_path`` fixture and other related ones provides resolved path (a.k.a real path)


- `4667 &lt;https://github.com/pytest-dev/pytest/issues/4667&gt;`_: ``pytest_terminal_summary`` uses result from ``pytest_report_teststatus`` hook, rather than hardcoded strings.


- `4669 &lt;https://github.com/pytest-dev/pytest/issues/4669&gt;`_: Correctly handle ``unittest.SkipTest`` exception containing non-ascii characters on Python 2.


- `4680 &lt;https://github.com/pytest-dev/pytest/issues/4680&gt;`_: Ensure the ``tmpdir`` and the ``tmp_path`` fixtures are the same folder.


- `4681 &lt;https://github.com/pytest-dev/pytest/issues/4681&gt;`_: Ensure ``tmp_path`` is always a real path.



Trivial/Internal Changes
------------------------

- `4643 &lt;https://github.com/pytest-dev/pytest/issues/4643&gt;`_: Use ``a.item()`` instead of the deprecated ``np.asscalar(a)`` in ``pytest.approx``.

  ``np.asscalar`` has been `deprecated &lt;https://github.com/numpy/numpy/blob/master/doc/release/1.16.0-notes.rstnew-deprecations&gt;`__ in ``numpy 1.16.``.


- `4657 &lt;https://github.com/pytest-dev/pytest/issues/4657&gt;`_: Copy saferepr from pylib
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest
  - Changelog: https://pyup.io/changelogs/pytest/
  - Homepage: https://docs.pytest.org/en/latest/
</details>





### Update [pytest-mock](https://pypi.org/project/pytest-mock) from **1.10.0** to **1.10.1**.


*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest-mock
  - Changelog: https://pyup.io/changelogs/pytest-mock/
  - Repo: https://github.com/pytest-dev/pytest-mock/
</details>





### Update [Sphinx](https://pypi.org/project/Sphinx) from **1.8.3** to **1.8.4**.


<details>
  <summary>Changelog</summary>
  
  
   ### 1.8.4
   ```
   =====================================

Bugs fixed
----------

* 3707: latex: no bold checkmark (✔) available.
* 5605: with the documentation language set to Chinese, English words could not
  be searched.
* 5889: LaTeX: user ``numfig_format`` is stripped of spaces and may cause
  build failure
* C++, fix hyperlinks for declarations involving east cv-qualifiers.
* 5755: C++, fix duplicate declaration error on function templates with constraints
  in the return type.
* C++, parse unary right fold expressions and binary fold expressions.
* pycode could not handle egg files on windows
* 5928: KeyError: &#39;DOCUTILSCONFIG&#39; when running build
* 5936: LaTeX: PDF build broken by inclusion of image taller than page height
  in an admonition
* 5231: &quot;make html&quot; does not read and build &quot;po&quot; files in &quot;locale&quot; dir
* 5954: ``:scale:`` image option may break PDF build if image in an admonition
* 5966: mathjax has not been loaded on incremental build
* 5960: LaTeX: modified PDF layout since September 2018 TeXLive update of
  :file:`parskip.sty`
* 5948: LaTeX: duplicated labels are generated for sections
* 5958: versionadded directive causes crash with Python 3.5.0
* 5995: autodoc: autodoc_mock_imports conflict with metaclass on Python 3.7
* 5871: texinfo: a section title ``.`` is not allowed
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/sphinx
  - Changelog: https://pyup.io/changelogs/sphinx/
  - Homepage: http://sphinx-doc.org/
</details>





### Update [pip](https://pypi.org/project/pip) from **18.1** to **19.0.1**.


<details>
  <summary>Changelog</summary>
  
  
   ### 19.0
   ```
   =================

Deprecations and Removals
-------------------------

- Deprecate support for Python 3.4 (`6106 &lt;https://github.com/pypa/pip/issues/6106&gt;`_)
- Start printing a warning for Python 2.7 to warn of impending Python 2.7 End-of-life and
  prompt users to start migrating to Python 3. (`6148 &lt;https://github.com/pypa/pip/issues/6148&gt;`_)
- Remove the deprecated ``--process-dependency-links`` option. (`6060 &lt;https://github.com/pypa/pip/issues/6060&gt;`_)
- Remove the deprecated SVN editable detection based on dependency links
  during freeze. (`5866 &lt;https://github.com/pypa/pip/issues/5866&gt;`_)

Features
--------

- Implement PEP 517 (allow projects to specify a build backend via pyproject.toml). (`5743 &lt;https://github.com/pypa/pip/issues/5743&gt;`_)
- Implement manylinux2010 platform tag support.  manylinux2010 is the successor
  to manylinux1.  It allows carefully compiled binary wheels to be installed
  on compatible Linux platforms. (`5008 &lt;https://github.com/pypa/pip/issues/5008&gt;`_)
- Improve build isolation: handle ``.pth`` files, so namespace packages are correctly supported under Python 3.2 and earlier. (`5656 &lt;https://github.com/pypa/pip/issues/5656&gt;`_)
- Include the package name in a freeze warning if the package is not installed. (`5943 &lt;https://github.com/pypa/pip/issues/5943&gt;`_)
- Warn when dropping an ``--[extra-]index-url`` value that points to an existing local directory. (`5827 &lt;https://github.com/pypa/pip/issues/5827&gt;`_)
- Prefix pip&#39;s ``--log`` file lines with their timestamp. (`6141 &lt;https://github.com/pypa/pip/issues/6141&gt;`_)

Bug Fixes
---------

- Avoid creating excessively long temporary paths when uninstalling packages. (`3055 &lt;https://github.com/pypa/pip/issues/3055&gt;`_)
- Redact the password from the URL in various log messages. (`4746 &lt;https://github.com/pypa/pip/issues/4746&gt;`_, `6124 &lt;https://github.com/pypa/pip/issues/6124&gt;`_)
- Avoid creating excessively long temporary paths when uninstalling packages. (`3055 &lt;https://github.com/pypa/pip/issues/3055&gt;`_)
- Avoid printing a stack trace when given an invalid requirement. (`5147 &lt;https://github.com/pypa/pip/issues/5147&gt;`_)
- Present 401 warning if username/password do not work for URL (`4833 &lt;https://github.com/pypa/pip/issues/4833&gt;`_)
- Handle ``requests.exceptions.RetryError`` raised in ``PackageFinder`` that was causing pip to fail silently when some indexes were unreachable. (`5270 &lt;https://github.com/pypa/pip/issues/5270&gt;`_, `5483 &lt;https://github.com/pypa/pip/issues/5483&gt;`_)
- Handle a broken stdout pipe more gracefully (e.g. when running ``pip list | head``). (`4170 &lt;https://github.com/pypa/pip/issues/4170&gt;`_)
- Fix crash from setting ``PIP_NO_CACHE_DIR=yes``. (`5385 &lt;https://github.com/pypa/pip/issues/5385&gt;`_)
- Fix crash from unparseable requirements when checking installed packages. (`5839 &lt;https://github.com/pypa/pip/issues/5839&gt;`_)
- Fix content type detection if a directory named like an archive is used as a package source. (`5838 &lt;https://github.com/pypa/pip/issues/5838&gt;`_)
- Fix listing of outdated packages that are not dependencies of installed packages in ``pip list --outdated --not-required`` (`5737 &lt;https://github.com/pypa/pip/issues/5737&gt;`_)
- Fix sorting ``TypeError`` in ``move_wheel_files()`` when installing some packages. (`5868 &lt;https://github.com/pypa/pip/issues/5868&gt;`_)
- Fix support for invoking pip using ``python src/pip ...``. (`5841 &lt;https://github.com/pypa/pip/issues/5841&gt;`_)
- Greatly reduce memory usage when installing wheels containing large files. (`5848 &lt;https://github.com/pypa/pip/issues/5848&gt;`_)
- Editable non-VCS installs now freeze as editable. (`5031 &lt;https://github.com/pypa/pip/issues/5031&gt;`_)
- Editable Git installs without a remote now freeze as editable. (`4759 &lt;https://github.com/pypa/pip/issues/4759&gt;`_)
- Canonicalize sdist file names so they can be matched to a canonicalized package name passed to ``pip install``. (`5870 &lt;https://github.com/pypa/pip/issues/5870&gt;`_)
- Properly decode special characters in SVN URL credentials. (`5968 &lt;https://github.com/pypa/pip/issues/5968&gt;`_)
- Make ``PIP_NO_CACHE_DIR`` disable the cache also for truthy values like ``&quot;true&quot;``, ``&quot;yes&quot;``, ``&quot;1&quot;``, etc. (`5735 &lt;https://github.com/pypa/pip/issues/5735&gt;`_)

Vendored Libraries
------------------

- Include license text of vendored 3rd party libraries. (`5213 &lt;https://github.com/pypa/pip/issues/5213&gt;`_)
- Update certifi to 2018.11.29
- Update colorama to 0.4.1
- Update distlib to 0.2.8
- Update idna to 2.8
- Update packaging to 19.0
- Update pep517 to 0.5.0
- Update pkg_resources to 40.6.3 (via setuptools)
- Update pyparsing to 2.3.1
- Update pytoml to 0.1.20
- Update requests to 2.21.0
- Update six to 1.12.0
- Update urllib3 to 1.24.1

Improved Documentation
----------------------

- Include the Vendoring Policy in the documentation. (`5958 &lt;https://github.com/pypa/pip/issues/5958&gt;`_)
- Add instructions for running pip from source to Development documentation. (`5949 &lt;https://github.com/pypa/pip/issues/5949&gt;`_)
- Remove references to removed ``egg=&lt;name&gt;-&lt;version&gt;`` functionality (`5888 &lt;https://github.com/pypa/pip/issues/5888&gt;`_)
- Fix omission of command name in HTML usage documentation (`5984 &lt;https://github.com/pypa/pip/issues/5984&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>







Co-authored-by: pyup-bot <github-bot@pyup.io>
Co-authored-by: Peter Bengtsson <mail@peterbe.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment