Don't explode sdists before installing them #166

woodruffw · 2021-12-02T20:28:15Z

Fixes #159.

woodruffw · 2021-12-02T20:31:24Z

This now fails with a ResolutionImpossible, which I'll also tackle in this PR:

resolvelib.resolvers.ResolutionImpossible:
[
  RequirementInformation(
    requirement=<Requirement('setuptools==57.4.0')>,
    parent=<progress==1.6 wheel=False>
  ),
  RequirementInformation(
    requirement=<Requirement('setuptools<51.0.0,>=50.3.2')>,
    parent=<cyclonedx-python-lib==0.11.1 wheel=True>
  ),
  RequirementInformation(
    requirement=<Requirement('setuptools==57.4.0')>,
    parent=<progress==1.6 wheel=False>
  ),
  RequirementInformation(
    requirement=<Requirement('setuptools==50.3.2')>,
    parent=<cyclonedx-python-lib==0.11.1 wheel=False>
  )
]

Off the top of my head, I'm, not sure why all of these dependencies have different versions of setuptools as subdependencies. Especially since only cyclonedx-python-lib lists it explicitly as a dependency.

woodruffw · 2021-12-02T21:17:32Z

Here's what happens when we install the cyclonedx-python-lib sdist:

DEBUG:pip_audit._virtual_env:completed: package_install_cmd=['/var/folders/zj/hy934vnj5xs68zv6w4b_f6s40000gn/T/tmpl7rquzwk/bin/python3', '-m', 'pip', 'install', '--no-cache-dir', '/var/folders/zj/hy934vnj5xs68zv6w4b_f6s40000gn/T/tmps27q9klr/cyclonedx-python-lib-0.11.1.tar.gz'], process.stdout=b"Processing /var/folders/zj/hy934vnj5xs68zv6w4b_f6s40000gn/T/tmps27q9klr/cyclonedx-python-lib-0.11.1.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
  Collecting toml<0.11.0,>=0.10.2
  Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
  Collecting setuptools<51.0.0,>=50.3.2
  Downloading setuptools-50.3.2-py3-none-any.whl (785 kB)
  Collecting types-toml<0.11.0,>=0.10.1
  Downloading types_toml-0.10.1-py3-none-any.whl (2.1 kB)
  Collecting requirements_parser<0.3.0,>=0.2.0
  Downloading requirements-parser-0.2.0.tar.gz (6.3 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
  Collecting packageurl-python<0.10.0,>=0.9.4
  Downloading packageurl_python-0.9.6-py3-none-any.whl (23 kB)
  Collecting types-setuptools<58.0.0,>=57.4.2
  Downloading types_setuptools-57.4.4-py3-none-any.whl (27 kB)
  Building wheels for collected packages: cyclonedx-python-lib, requirements-parser
  Building wheel for cyclonedx-python-lib (pyproject.toml): started
  Building wheel for cyclonedx-python-lib (pyproject.toml): finished with status 'done'
  Created wheel for cyclonedx-python-lib: filename=cyclonedx_python_lib-0.11.1-py3-none-any.whl size=127708 sha256=ed6cf44f29b1f31835f818ca3b18deaee28be32a3179eadf9ad6c853217c93f9
  Stored in directory: /private/var/folders/zj/hy934vnj5xs68zv6w4b_f6s40000gn/T/pip-ephem-wheel-cache-x66n48lz/wheels/df/1c/0a/ab8052acc2f11b332db5a7d778754ce1f2da723056cef5d7d5
  Building wheel for requirements-parser (setup.py): started
  Building wheel for requirements-parser (setup.py): finished with status 'done'
  Created wheel for requirements-parser: filename=requirements_parser-0.2.0-py3-none-any.whl size=7688 sha256=08fe0acc14dda5c973036b3cd6c17e4870f701f748c6bdc1d3a21205e9a48630
  Stored in directory: /private/var/folders/zj/hy934vnj5xs68zv6w4b_f6s40000gn/T/pip-ephem-wheel-cache-x66n48lz/wheels/97/3a/c8/0a5a6ac041c7b83638f97b4a69b2cf6e76a03a1643d3926f5b
  Successfully built cyclonedx-python-lib requirements-parser
Installing collected packages: types-toml, types-setuptools, toml, setuptools, requirements-parser, packageurl-python, cyclonedx-python-lib
  Attempting uninstall: setuptools
    Found existing installation: setuptools 57.4.0
    Uninstalling setuptools-57.4.0:
      Successfully uninstalled setuptools-57.4.0
      Successfully installed cyclonedx-python-lib-0.11.1 packageurl-python-0.9.6 requirements-parser-0.2.0 setuptools-50.3.2 toml-0.10.2 types-setuptools-57.4.4 types-toml-0.10.1
", process.stderr=b''

So it downgrades the already installed setuptools, setting up the conflict with progress. I'm not really sure why it does that, given that the newer version is compatible with the version range and we haven't asked it to downgrade.

woodruffw · 2021-12-02T21:50:14Z

Being a little more clever about sdist filtering reduced the resolution error down to two:

[
    RequirementInformation(
        requirement=<Requirement('setuptools==57.4.0')>,
        parent=<progress==1.6 wheel=False>
    ),
    RequirementInformation(
        requirement=<Requirement('setuptools<51.0.0,>=50.3.2')>,
        parent=<cyclonedx-python-lib==0.11.1 wheel=True>
    )
]

di · 2021-12-02T21:54:07Z

   RequirementInformation(
       requirement=<Requirement('setuptools==57.4.0')>,
       parent=<progress==1.6 wheel=False>
   ),

Where is this coming from? That's a really tight pin that I don't see anywhere in progress's source distribution for that release

woodruffw · 2021-12-02T21:59:20Z

Where is this coming from?

That's a really good question; I'm not 100% sure yet.

Our install step for progress boils down to python -m pip install progress-1.6.tar.gz inside of a virtual environment, so we shouldn't be doing anything particularly unusual.

Here's the pip install step as it runs:

DEBUG:pip_audit._virtual_env:ran: ['/var/folders/zj/hy934vnj5xs68zv6w4b_f6s40000gn/T/tmpadmaoxgu/bin/python3', '-m', 'pip', 'install', 'progress==1.6'], stdout=b"Collecting progress==1.6
  Downloading progress-1.6.tar.gz (7.8 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
  Building wheels for collected packages: progress
  Building wheel for progress (setup.py): started
  Building wheel for progress (setup.py): finished with status 'done'
  Created wheel for progress: filename=progress-1.6-py3-none-any.whl size=9628 sha256=6c6ea87b7ca81e99c45aa07897185d02aa948cd8d0d1bc15446fd2164d0bd21c
  Stored in directory: /Users/william/Library/Caches/pip/wheels/a2/68/5f/c339b20a41659d856c93ccdce6a33095493eb82c3964aac5a1
  Successfully built progress
Installing collected packages: progress
Successfully installed progress-1.6
", stderr=b''

woodruffw · 2021-12-03T22:23:16Z

This is ready for review: it doesn't fix the impossible resolution bug, but I want to get it out of the way so that we can make other (performance-minded) improvements to this part of the code.

tetsuo-cpp

LGTM!

tetsuo-cpp · 2021-12-05T02:57:23Z

pip_audit/_dependency_source/resolvelib/pypi_provider.py

-                names = t.getnames()
-                pkg_name = names[0]
-                t.extractall(pkg_dir)
+            sdist = Path(pkg_dir) / self.filename.name


Haha that's awesome. I didn't realise you could do that.

## [1.1.0] ### Added * CLI: The `--path <PATH>` flag has been added, allowing users to limit dependency discovery to one or more paths (specified separately) when `pip-audit` is invoked in environment mode ([#148](pypa/pip-audit#148)) * CLI: The `pip-audit` CLI can now be accessed through `python -m pip_audit`. All functionality is identical to the functionality provided by the `pip-audit` entrypoint ([#173](pypa/pip-audit#173)) * CLI: The `--verbose` flag has been added, allowing users to receive more more verbose output from `pip-audit`. Supplying the `--verbose` flag overrides the `PIP_AUDIT_LOGLEVEL` environment variable and is equivalent to setting it to `debug` ([#185](pypa/pip-audit#185)) ### Changed * CLI: `pip-audit` now clears its spinner bar from the terminal upon completion, preventing visual confusion ([#174](pypa/pip-audit#174)) ### Fixed * Dependency sources: a crash caused by `platform.python_version` returning an version string that couldn't be parsed as a PEP-440 version was fixed ([#175](pypa/pip-audit#175)) * Dependency sources: a crash caused by incorrect assumptions about the structure of source distributions was fixed ([#166](pypa/pip-audit#166)) * Vulnerability sources: a performance issue on Windows caused by cache failures was fixed ([#178](pypa/pip-audit#178)) ## [1.0.1] - 2021-12-02 ### Fixed * CLI: The `--desc` flag no longer requires a following argument. If passed as a bare option, `--desc` is equivalent to `--desc on` ([#153](pypa/pip-audit#153)) * Dependency resolution: The PyPI-based dependency resolver no longer throws an uncaught exception on package resolution errors; instead, the package is marked as skipped and an appropriate warning or fatal error (in `--strict` mode) is produced ([#162](pypa/pip-audit#162)) * CLI: When providing the `--cache-dir` flag, the command to read the pip cache directory is no longer executed. Previously this was always executed and could result into failure when the command fails. In CI environments, the default `~/.cache` directory is typically not writable by the build user and this meant that the `python -m pip cache dir` would fail before this fix, even if the `--cache-dir` flag was provided. ([#161](pypa/pip-audit#161)) ## [1.0.0] - 2021-12-01 ### Added * This is the first stable release of `pip-audit`! The CLI is considered stable from this point on, and all changes will comply with [Semantic Versioning](https://semver.org/) ## [0.0.9] - 2021-12-01 ### Added * CLI: Skipped dependencies are now listed in the output of `pip-audit`, for supporting output formats ([#145](pypa/pip-audit#145)) * CLI: `pip-audit` now supports a "strict" mode (enabled with `-S` or `--strict`) that fails if the audit if any individual dependency cannot be resolved or audited. The default behavior is still to skip any individual dependency errors ([#146](pypa/pip-audit#146))

woodruffw added 2 commits December 2, 2021 15:26

pip_audit: don't unpack sdists before installing

770187c

Merge remote-tracking branch 'origin/main' into ww/dist-bug

379bb5f

woodruffw added bug Something isn't working component:dep-sources Dependency sources labels Dec 2, 2021

woodruffw self-assigned this Dec 2, 2021

pypi_provider: simplify sdist placement

5fa322a

woodruffw added 2 commits December 2, 2021 16:45

pip_audit: logging, keep sdists out of candidacy

e58e9a1

pypi_provider: remove old code

e6c60b3

_virtual_env: fix f-strings

b70481d

_virtual_env: re-enable cache

ec61396

woodruffw added 2 commits December 3, 2021 16:55

Merge remote-tracking branch 'origin/main' into ww/dist-bug

303fc49

_virtual_env: remove log

272240c

woodruffw marked this pull request as ready for review December 3, 2021 22:22

woodruffw requested review from tetsuo-cpp and di December 3, 2021 22:22

woodruffw added the performance Something isn't as fast or responsive as it should be. label Dec 3, 2021

CHANGELOG: record changes

a628f31

di approved these changes Dec 4, 2021

View reviewed changes

tetsuo-cpp added 2 commits December 4, 2021 15:53

Merge branch 'main' into ww/dist-bug

f6bddb2

_virtual_env: Update comment

3ce15e9

tetsuo-cpp approved these changes Dec 5, 2021

View reviewed changes

tetsuo-cpp merged commit a8f9fa2 into main Dec 5, 2021

tetsuo-cpp deleted the ww/dist-bug branch December 5, 2021 03:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Don't explode sdists before installing them #166

Don't explode sdists before installing them #166

woodruffw commented Dec 2, 2021 •

edited

woodruffw commented Dec 2, 2021 •

edited

woodruffw commented Dec 2, 2021

woodruffw commented Dec 2, 2021

di commented Dec 2, 2021

woodruffw commented Dec 2, 2021 •

edited

woodruffw commented Dec 3, 2021

tetsuo-cpp left a comment

tetsuo-cpp Dec 5, 2021

Don't explode sdists before installing them #166

Don't explode sdists before installing them #166

Conversation

woodruffw commented Dec 2, 2021 • edited

woodruffw commented Dec 2, 2021 • edited

woodruffw commented Dec 2, 2021

woodruffw commented Dec 2, 2021

di commented Dec 2, 2021

woodruffw commented Dec 2, 2021 • edited

woodruffw commented Dec 3, 2021

tetsuo-cpp left a comment

Choose a reason for hiding this comment

tetsuo-cpp Dec 5, 2021

Choose a reason for hiding this comment

woodruffw commented Dec 2, 2021 •

edited

woodruffw commented Dec 2, 2021 •

edited

woodruffw commented Dec 2, 2021 •

edited