Dependency resolution fails to install `cyclonedx-python-lib-0.11.1` #159

tetsuo-cpp · 2021-12-02T07:39:00Z

Bug description

If I audit a requirements file that transitively depends on cyclonedx-python-lib, I see a pip install failure.

Reproduction steps

Make a requirements.txt that looks like this (or any dependency that relies on cyclonedx-python-lib):

pip-audit==1.0.0

Run pip-audit -r requirements.txt
Observe the failure

Expected behavior

All dependencies are audited properly.

Screenshots and logs

(.ve) tetsuo@Alexs-MacBook-Pro test % pip-audit -r requirements.txt
- Installing package in isolated environment
Traceback (most recent call last):
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_virtual_env.py", line 90, in post_setup
    subprocess.run(
  File "/opt/homebrew/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/var/folders/mq/25w5gx6x0w7c5hq2l6tz8k_40000gn/T/tmppf9hai1y/bin/python', '-m', 'pip', 'install', '-e', '/var/folders/mq/25w5gx6x0w7c5hq2l6tz8k_40000gn/T/tmpykqq2afm/cyclonedx-python-lib-0.11.1/LICENSE']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/tetsuo/Workspace/test/.ve/bin/pip-audit", line 8, in <module>
    sys.exit(audit())
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_cli.py", line 240, in audit
    for (spec, vulns) in auditor.audit(source):
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_audit.py", line 60, in audit
    yield from self._service.query_all(specs)
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_service/interface.py", line 115, in query_all
    for spec in specs:
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/requirement.py", line 62, in collect
    for _, deps in self.resolver.resolve_all(iter(req_values)):
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/interface.py", line 67, in resolve_all
    yield (req, self.resolve(req))
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/resolvelib/resolvelib.py", line 41, in resolve
    result = self.resolver.resolve([req])
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/resolvelib/resolvers.py", line 373, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/resolvelib/resolvers.py", line 203, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 293, in get_dependencies
    return candidate.dependencies
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 112, in dependencies
    self._dependencies = list(self._get_dependencies())
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 94, in _get_dependencies
    deps: List[str] = self.metadata.get_all("Requires-Dist", [])
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 87, in metadata
    self._metadata = self._get_metadata_for_sdist()
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 170, in _get_metadata_for_sdist
    ve.create(ve_dir)
  File "/opt/homebrew/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/venv/__init__.py", line 78, in create
    self.post_setup(context)
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_virtual_env.py", line 97, in post_setup
    raise VirtualEnvError(f"Failed to install packages: {package_install_cmd}") from cpe
pip_audit._virtual_env.VirtualEnvError: Failed to install packages: ['/var/folders/mq/25w5gx6x0w7c5hq2l6tz8k_40000gn/T/tmppf9hai1y/bin/python', '-m', 'pip', 'install', '-e', '/var/folders/mq/25w5gx6x0w7c5hq2l6tz8k_40000gn/T/tmpykqq2afm/cyclonedx-python-lib-0.11.1/LICENSE']
(.ve) tetsuo@Alexs-MacBook-Pro test % vim requirements.txt
(.ve) tetsuo@Alexs-MacBook-Pro test % PIP_AUDIT_LOGLEVEL=debug pip-audit -r requirements.txt
DEBUG:pip_audit._cli:parsed arguments: Namespace(local=False, requirements=[<_io.TextIOWrapper name='requirements.txt' mode='r' encoding='UTF-8'>], format=<OutputFormatChoice.Columns: 'columns'>, vulnerability_service=<VulnerabilityServiceChoice.Pypi: 'pypi'>, dry_run=False, strict=False, desc=<VulnerabilityDescriptionChoice.Auto: 'auto'>, cache_dir=None, progress_spinner=<ProgressSpinnerChoice.On: 'on'>, timeout=15)
- Installing package in isolated environment
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/pip-audit HTTP/1.1" 301 215
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/pip-audit/ HTTP/1.1" 200 2843
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): files.pythonhosted.org:443
DEBUG:urllib3.connectionpool:https://files.pythonhosted.org:443 "GET /packages/16/b2/c8a8208030b89825cdc1a4f2edaabf2076ba38ab07e77213fc299cf74d99/pip_audit-1.0.0-py3-none-any.whl HTTP/1.1" 200 49177
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/pip-api HTTP/1.1" 301 213
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/pip-api/ HTTP/1.1" 200 4058
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/packaging HTTP/1.1" 301 215
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/packaging/ HTTP/1.1" 200 7379
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/progress HTTP/1.1" 301 214
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/progress/ HTTP/1.1" 200 1044
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/resolvelib HTTP/1.1" 301 216
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/resolvelib/ HTTP/1.1" 200 3179
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/html5lib HTTP/1.1" 301 214
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/html5lib/ HTTP/1.1" 200 3097
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/cachecontrol HTTP/1.1" 301 218
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/cachecontrol/ HTTP/1.1" 200 4358
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/lockfile HTTP/1.1" 301 214
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/lockfile/ HTTP/1.1" 200 990
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/cyclonedx-python-lib HTTP/1.1" 301 226
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/cyclonedx-python-lib/ HTTP/1.1" 200 4903
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): files.pythonhosted.org:443
DEBUG:urllib3.connectionpool:https://files.pythonhosted.org:443 "GET /packages/2a/68/d8412d1e0d70edf9791cbac5426dc859f4649afc22f2abbeb0d947cf70fd/progress-1.6.tar.gz HTTP/1.1" 200 7842
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/pip HTTP/1.1" 301 109
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/pip/ HTTP/1.1" 200 18015
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/progress HTTP/1.1" 301 214
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/progress/ HTTP/1.1" 200 1044
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/setuptools HTTP/1.1" 301 216
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/setuptools/ HTTP/1.1" 200 104565
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): files.pythonhosted.org:443
DEBUG:urllib3.connectionpool:https://files.pythonhosted.org:443 "GET /packages/e8/28/b6b91e2dd3dbc964db396650703b922d5f0047a32c7f1af5c3c6c662b5cb/pip_api-0.0.23-py3-none-any.whl HTTP/1.1" 200 107323
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/pip HTTP/1.1" 301 109
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/pip/ HTTP/1.1" 200 18015
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): files.pythonhosted.org:443
DEBUG:urllib3.connectionpool:https://files.pythonhosted.org:443 "GET /packages/6c/dd/a834df6482147d48e225a49515aabc28974ad5a4ca3215c18a882565b028/html5lib-1.1-py2.py3-none-any.whl HTTP/1.1" 200 112173
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/six HTTP/1.1" 301 109
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/six/ HTTP/1.1" 200 4429
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/webencodings HTTP/1.1" 301 218
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/webencodings/ HTTP/1.1" 200 891
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): files.pythonhosted.org:443
DEBUG:urllib3.connectionpool:https://files.pythonhosted.org:443 "GET /packages/d3/39/b7cd9ef1be03ac33e71f76837a23d59842b016e5159cf5aff30c0b340907/CacheControl-0.12.10-py2.py3-none-any.whl HTTP/1.1" 200 20297
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/requests HTTP/1.1" 301 114
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/requests/ HTTP/1.1" 200 18544
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/msgpack HTTP/1.1" 301 213
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/msgpack/ HTTP/1.1" 200 28426
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): files.pythonhosted.org:443
DEBUG:urllib3.connectionpool:https://files.pythonhosted.org:443 "GET /packages/c8/22/9460e311f340cb62d26a38c419b1381b8593b0bb6b5d1f056938b086d362/lockfile-0.12.2-py2.py3-none-any.whl HTTP/1.1" 200 13564
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): files.pythonhosted.org:443
DEBUG:urllib3.connectionpool:https://files.pythonhosted.org:443 "GET /packages/0c/51/51b9438d2813d8252cade62650347b38fbf692e761cc0511d8c81adfdd31/cyclonedx_python_lib-0.11.1-py3-none-any.whl HTTP/1.1" 200 127708
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/packageurl-python HTTP/1.1" 301 223
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/packageurl-python/ HTTP/1.1" 200 3928
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/requirements-parser HTTP/1.1" 301 225
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/requirements-parser/ HTTP/1.1" 200 822
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org:443
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/setuptools HTTP/1.1" 301 216
DEBUG:urllib3.connectionpool:https://pypi.org:443 "GET /simple/setuptools/ HTTP/1.1" 200 104565
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): files.pythonhosted.org:443
DEBUG:urllib3.connectionpool:https://files.pythonhosted.org:443 "GET /packages/13/ea/5720270a0e984e5dd238ec8e75a8acccb042ac97bcb215be9e2cb339dc1f/cyclonedx-python-lib-0.11.1.tar.gz HTTP/1.1" 200 98048
Traceback (most recent call last):
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_virtual_env.py", line 90, in post_setup
    subprocess.run(
  File "/opt/homebrew/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/var/folders/mq/25w5gx6x0w7c5hq2l6tz8k_40000gn/T/tmpapwlw08o/bin/python', '-m', 'pip', 'install', '-e', '/var/folders/mq/25w5gx6x0w7c5hq2l6tz8k_40000gn/T/tmpz7uf8d_w/cyclonedx-python-lib-0.11.1/LICENSE']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/tetsuo/Workspace/test/.ve/bin/pip-audit", line 8, in <module>
    sys.exit(audit())
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_cli.py", line 240, in audit
    for (spec, vulns) in auditor.audit(source):
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_audit.py", line 60, in audit
    yield from self._service.query_all(specs)
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_service/interface.py", line 115, in query_all
    for spec in specs:
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/requirement.py", line 62, in collect
    for _, deps in self.resolver.resolve_all(iter(req_values)):
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/interface.py", line 67, in resolve_all
    yield (req, self.resolve(req))
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/resolvelib/resolvelib.py", line 41, in resolve
    result = self.resolver.resolve([req])
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/resolvelib/resolvers.py", line 481, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/resolvelib/resolvers.py", line 373, in resolve
    failure_causes = self._attempt_to_pin_criterion(name)
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
    criteria = self._get_updated_criteria(candidate)
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/resolvelib/resolvers.py", line 203, in _get_updated_criteria
    for requirement in self._p.get_dependencies(candidate=candidate):
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 293, in get_dependencies
    return candidate.dependencies
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 112, in dependencies
    self._dependencies = list(self._get_dependencies())
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 94, in _get_dependencies
    deps: List[str] = self.metadata.get_all("Requires-Dist", [])
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 87, in metadata
    self._metadata = self._get_metadata_for_sdist()
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 170, in _get_metadata_for_sdist
    ve.create(ve_dir)
  File "/opt/homebrew/Cellar/python@3.9/3.9.7_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/venv/__init__.py", line 78, in create
    self.post_setup(context)
  File "/Users/tetsuo/Workspace/test/.ve/lib/python3.9/site-packages/pip_audit/_virtual_env.py", line 97, in post_setup
    raise VirtualEnvError(f"Failed to install packages: {package_install_cmd}") from cpe
pip_audit._virtual_env.VirtualEnvError: Failed to install packages: ['/var/folders/mq/25w5gx6x0w7c5hq2l6tz8k_40000gn/T/tmpapwlw08o/bin/python', '-m', 'pip', 'install', '-e', '/var/folders/mq/25w5gx6x0w7c5hq2l6tz8k_40000gn/T/tmpz7uf8d_w/cyclonedx-python-lib-0.11.1/LICENSE']

Platform information

OS name and version: macOS 12.0.1
pip-audit version (pip-audit -V): 1.0.0
Python version (python -V or python3 -V): 3.9.7
pip version (pip -V or pip3 -V): 21.3.1

The text was updated successfully, but these errors were encountered:

tetsuo-cpp · 2021-12-02T07:40:47Z

We don't seem to be getting the pip install invocation correct. It's passing in the LICENSE file for some reason?

Perhaps something about the directory structure of this package is different.

woodruffw · 2021-12-02T15:45:26Z

Huh, that's concerning. Looking into it.

Edit: Forgot to say, I was also able to repro this locally.

woodruffw · 2021-12-02T15:48:01Z

First step: it's not an sdist issue, since cyclonedx-python-lib thankfully provides a wheel: https://files.pythonhosted.org/packages/0c/51/51b9438d2813d8252cade62650347b38fbf692e761cc0511d8c81adfdd31/cyclonedx_python_lib-0.11.1-py3-none-any.whl

So my expectation is that we'd pass that .whl file directly to pip install, since that should "just work". Checking that now.

woodruffw · 2021-12-02T15:50:50Z

Actually, this is making even less sense: we shouldn't be calling pip install at all for a wheel, since all of the metadata is self-contained and doesn't require us to perform an installation. So our sdist/bdist code is being confused somewhere.

woodruffw · 2021-12-02T16:19:14Z

Okay, here's the candidate selection step's results:

DEBUG:pip_audit._dependency_source.resolvelib.pypi_provider:candidates: [[<cy
clonedx-python-lib==0.11.1 url=https://files.pythonhosted.org/packages/0c/51/51b9438d2813d8252cade62650347b38fbf692e761cc0511d8c81adfdd31/cyclonedx_python_lib-0.11.1-py3-none-any.whl#sha256=ed6cf44f29b1f31835f818ca3b18deaee28be32a3179eadf9ad6c853217c93f9 wheel=True>, <cyclonedx-python-lib==0.11.1 url=https://files.pythonhosted.org/packages/13/ea/5720270a0e984e5dd238ec8e75a8acccb042ac97bcb215be9e2cb339dc1f/cyclonedx-python-lib-0.11.1.tar.gz#sha256=cb0f1730ebe23c37820a9a2d4b42fc1d19fb3e8e6e92dfd3489673c76152e43c wheel=False>]]

So that all looks correct: the wheel candidate is correctly identified as a wheel, and the sdist is correctly identified as a non-wheel.

woodruffw · 2021-12-02T16:25:52Z

New theory: this is actually an uninitialized Candidate state issue. The traceback shows that we go from PyPIProvider.get_dependencies() to Candidate.dependencies, which calls Candidate._get_dependencies() internally.

That in turn accesses self.metadata, which in turn is a property-style-function that checks is_wheel to determine the metadata technique to use. Candidate.is_wheel defaults to True, so it's possible that we have a partially initialized Candidate whose metadata is being accessed. But that also isn't obvious from the code, since the only place where we generate Candidate objects is get_project_from_pypi.

woodruffw · 2021-12-02T17:20:30Z

I'm still not clear on why the source distribution install fails, but I think I know why it's happening now:

We collect all candidates for cyclonedx-python-lib
We sort them by version and is_wheel, such that our top two candidates are (1) the wheel, and (2) the sdist for the current version (0.11.1)

resolvelib's Resolution object tries to call _attempt_to_pin_criterion("cyclonedx-python-lib"), which does this:

      for candidate in criterion.candidates:
          try:
              criteria = self._get_updated_criteria(candidate)
          except RequirementsConflicted as e:
              causes.append(e.criterion)
              continue

...where _get_updated_criteria makes the _get_dependencies call that breaks everything:

   def _get_updated_criteria(self, candidate):
       criteria = self.state.criteria.copy()
       for requirement in self._p.get_dependencies(candidate=candidate):
           self._add_to_criteria(criteria, requirement, parent=candidate)
       return criteria

So, there are two problems here:

resolvelib tries to eagerly fetch the dependencies for both candidates, which in turn means that we end up wasting a lot of time setting up a local virtual environment that's effectively identical in state to the wheel;
There's still some bug either in resolvelib or our code that's causing us to try to install a wheel as if it's a source distribution.

woodruffw · 2021-12-02T17:37:31Z

Ugh. Problem (2) really is a malformed-ish tarball issue.

Here's what the listing for progress's tarball looks like:

drwxr-xr-x  0 verigak staff       0 Jul 28 02:52 progress-1.6/
-rw-r--r--  0 verigak staff    5208 Jul 28 02:52 progress-1.6/PKG-INFO
-rw-r--r--  0 verigak staff     777 Jul 20  2020 progress-1.6/LICENSE
drwxr-xr-x  0 verigak staff       0 Jul 28 02:52 progress-1.6/progress/
-rw-r--r--  0 verigak staff    5294 Jul 28 02:50 progress-1.6/progress/__init__.py
-rw-r--r--  0 verigak staff    2942 Jul 20  2020 progress-1.6/progress/bar.py
-rw-r--r--  0 verigak staff    1461 Jul 20  2020 progress-1.6/progress/spinner.py
-rw-r--r--  0 verigak staff    1613 Jul 20  2020 progress-1.6/progress/counter.py
-rw-r--r--  0 verigak staff    2655 Jul 20  2020 progress-1.6/progress/colors.py
-rw-r--r--  0 verigak staff      45 Feb  1  2019 progress-1.6/MANIFEST.in
-rwxr-xr-x  0 verigak staff    1638 Jul 28 02:49 progress-1.6/test_progress.py
-rwxr-xr-x  0 verigak staff     844 Jul 20  2020 progress-1.6/setup.py
-rw-r--r--  0 verigak staff      38 Jul 28 02:52 progress-1.6/setup.cfg
drwxr-xr-x  0 verigak staff       0 Jul 28 02:52 progress-1.6/progress.egg-info/
-rw-r--r--  0 verigak staff    5208 Jul 28 02:52 progress-1.6/progress.egg-info/PKG-INFO
-rw-r--r--  0 verigak staff     280 Jul 28 02:52 progress-1.6/progress.egg-info/SOURCES.txt
-rw-r--r--  0 verigak staff       9 Jul 28 02:52 progress-1.6/progress.egg-info/top_level.txt
-rw-r--r--  0 verigak staff       1 Jul 28 02:52 progress-1.6/progress.egg-info/dependency_links.txt
-rw-r--r--  0 verigak staff    3319 Feb  1  2019 progress-1.6/README.rst

...and here's what cyclonedx-python-lib's looks like:

-rw-r--r--  0 0      0       11357 Nov 10 10:41 cyclonedx-python-lib-0.11.1/LICENSE
-rw-r--r--  0 0      0        9622 Nov 10 10:41 cyclonedx-python-lib-0.11.1/README.md
-rw-r--r--  0 0      0         697 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/__init__.py
-rw-r--r--  0 0      0         732 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/exception/__init__.py
-rw-r--r--  0 0      0         944 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/exception/parser.py
-rw-r--r--  0 0      0        6692 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/model/__init__.py
-rw-r--r--  0 0      0        8619 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/model/bom.py
-rw-r--r--  0 0      0        9721 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/model/component.py
-rw-r--r--  0 0      0        9429 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/model/vulnerability.py
-rw-r--r--  0 0      0        2932 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/output/__init__.py
-rw-r--r--  0 0      0        5338 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/output/json.py
-rw-r--r--  0 0      0        2249 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/output/schema.py
-rw-r--r--  0 0      0       11179 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/output/xml.py
-rw-r--r--  0 0      0        4699 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/parser/__init__.py
-rw-r--r--  0 0      0        3419 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/parser/conda.py
-rw-r--r--  0 0      0        2776 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/parser/environment.py
-rw-r--r--  0 0      0        2272 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/parser/pipenv.py
-rw-r--r--  0 0      0        2200 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/parser/poetry.py
-rw-r--r--  0 0      0        2365 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/parser/requirements.py
-rw-r--r--  0 0      0         153 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/py.typed
-rw-r--r--  0 0      0       13619 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/bom-1.0.xsd
-rw-r--r--  0 0      0       39484 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/bom-1.1.xsd
-rw-r--r--  0 0      0       36866 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/bom-1.2-strict.schema.json
-rw-r--r--  0 0      0       35999 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/bom-1.2.schema.json
-rw-r--r--  0 0      0       76147 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/bom-1.2.xsd
-rw-r--r--  0 0      0       40019 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/bom-1.3-strict.schema.json
-rw-r--r--  0 0      0       23176 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/bom-1.3.proto
-rw-r--r--  0 0      0       39090 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/bom-1.3.schema.json
-rw-r--r--  0 0      0       88384 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/bom-1.3.xsd
-rw-r--r--  0 0      0        9063 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/ext/bom-descriptor-0.9.xsd
-rw-r--r--  0 0      0        8233 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/ext/bom-descriptor-1.0.xsd
-rw-r--r--  0 0      0        3146 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/ext/dependency-graph-1.0.xsd
-rw-r--r--  0 0      0        6380 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/ext/vulnerability-1.0-SNAPSHOT.schema.json
-rw-r--r--  0 0      0       14195 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/ext/vulnerability-1.0.xsd
-rw-r--r--  0 0      0        9451 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/spdx.schema.json
-rw-r--r--  0 0      0      113598 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/schema/spdx.xsd
-rw-r--r--  0 0      0           0 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/utils/__init__.py
-rw-r--r--  0 0      0        4127 Nov 10 10:41 cyclonedx-python-lib-0.11.1/cyclonedx/utils/conda.py
-rw-r--r--  0 0      0        2281 Nov 10 10:41 cyclonedx-python-lib-0.11.1/pyproject.toml
-rw-r--r--  0 0      0       11172 Nov 10 10:42 cyclonedx-python-lib-0.11.1/setup.py
-rw-r--r--  0 0      0       11490 Nov 10 10:42 cyclonedx-python-lib-0.11.1/PKG-INFO

So there's no separate archive record for the directory, which is what we expected. Groan. I'll see what other packaging tools do to handle this.

di · 2021-12-02T18:01:10Z

Seems like our logic is bad here:

https://github.com/trailofbits/pip-audit/blob/5216e4aa61ff5e482d300d0bc0e7eb57ba9e7f82/pip_audit/_dependency_source/resolvelib/pypi_provider.py#L154-L156

I think we should get the package name from the metadata instead, something like https://github.com/pypa/pip/blob/4fac2b90a5d200b46e7b576013bb25f4ebb3f937/src/pip/_internal/utils/packaging.py#L41-L56.

woodruffw · 2021-12-02T18:03:11Z

I think we should get the package name from the metadata instead, something like https://github.com/pypa/pip/blob/4fac2b90a5d200b46e7b576013bb25f4ebb3f937/src/pip/_internal/utils/packaging.py#L41-L56.

We should, but it's a little circular: we need to unpack before we can get to the metadata. pip takes a different approach to unpacking than we do to avoid this problem; I'm currently working on a PR that adapts their approach.

hschilling · 2021-12-04T17:00:18Z

FYI, I am also getting a similar error with the attached requirements file.

requirements.txt


pip-audit -r ./requirements.txt
\ Installing package in isolated environment
Traceback (most recent call last):
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/pip_audit/_virtual_env.py", line 90, in post_setup
   subprocess.run(
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/subprocess.py", line 512, in run
   raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/var/folders/1r/4gh355fx68ldr7pwx1b97gd00000gp/T/tmpz2w69r4i/bin/python', '-m', 'pip', 'install', '-e', '/var/folders/1r/4gh355fx68ldr7pwx1b97gd00000gp/T/tmpzotwdwuk/petsc4py-3.13.0']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
 File "/Users/hschilli/anaconda/envs/py38mpi4/bin/pip-audit", line 8, in <module>
   sys.exit(audit())
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/pip_audit/_cli.py", line 242, in audit
   for (spec, vulns) in auditor.audit(source):
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/pip_audit/_audit.py", line 60, in audit
   yield from self._service.query_all(specs)
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/pip_audit/_service/interface.py", line 115, in query_all
   for spec in specs:
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/pip_audit/_dependency_source/requirement.py", line 63, in collect
   for _, deps in self.resolver.resolve_all(iter(req_values)):
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/pip_audit/_dependency_source/interface.py", line 67, in resolve_all
   yield (req, self.resolve(req))
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/resolvelib.py", line 44, in resolve
   result = self.resolver.resolve([req])
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/resolvelib/resolvers.py", line 481, in resolve
   state = resolution.resolve(requirements, max_rounds=max_rounds)
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/resolvelib/resolvers.py", line 373, in resolve
   failure_causes = self._attempt_to_pin_criterion(name)
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/resolvelib/resolvers.py", line 213, in _attempt_to_pin_criterion
   criteria = self._get_updated_criteria(candidate)
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/resolvelib/resolvers.py", line 203, in _get_updated_criteria
   for requirement in self._p.get_dependencies(candidate=candidate):
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 295, in get_dependencies
   return candidate.dependencies
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 112, in dependencies
   self._dependencies = list(self._get_dependencies())
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 94, in _get_dependencies
   deps: List[str] = self.metadata.get_all("Requires-Dist", [])
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 87, in metadata
   self._metadata = self._get_metadata_for_sdist()
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/pip_audit/_dependency_source/resolvelib/pypi_provider.py", line 170, in _get_metadata_for_sdist
   ve.create(ve_dir)
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/venv/__init__.py", line 71, in create
   self.post_setup(context)
 File "/Users/hschilli/anaconda/envs/py38mpi4/lib/python3.8/site-packages/pip_audit/_virtual_env.py", line 97, in post_setup
   raise VirtualEnvError(f"Failed to install packages: {package_install_cmd}") from cpe
pip_audit._virtual_env.VirtualEnvError: Failed to install packages: ['/var/folders/1r/4gh355fx68ldr7pwx1b97gd00000gp/T/tmpz2w69r4i/bin/python', '-m', 'pip', 'install', '-e', '/var/folders/1r/4gh355fx68ldr7pwx1b97gd00000gp/T/tmpzotwdwuk/petsc4py-3.13.0']

di · 2021-12-04T22:45:40Z

@hschilling That seems unrelated, can you open a new issue?

hschilling · 2021-12-04T22:49:19Z

Sorry, I see what you mean. Will do!

Thanks for creating pip-audit

tetsuo-cpp added bug Something isn't working component:dep-sources Dependency sources labels Dec 2, 2021

woodruffw mentioned this issue Dec 2, 2021

Don't explode sdists before installing them #166

Merged

tetsuo-cpp closed this as completed in #166 Dec 5, 2021

woodruffw mentioned this issue Dec 7, 2021

ResolutionImpossible errors in requirements mode #197

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dependency resolution fails to install `cyclonedx-python-lib-0.11.1` #159

Dependency resolution fails to install `cyclonedx-python-lib-0.11.1` #159

tetsuo-cpp commented Dec 2, 2021 •

edited

Loading

tetsuo-cpp commented Dec 2, 2021 •

edited

Loading

woodruffw commented Dec 2, 2021 •

edited

Loading

woodruffw commented Dec 2, 2021

woodruffw commented Dec 2, 2021

woodruffw commented Dec 2, 2021

woodruffw commented Dec 2, 2021

woodruffw commented Dec 2, 2021

woodruffw commented Dec 2, 2021

di commented Dec 2, 2021

woodruffw commented Dec 2, 2021

hschilling commented Dec 4, 2021 •

edited

Loading

di commented Dec 4, 2021

hschilling commented Dec 4, 2021

Dependency resolution fails to install cyclonedx-python-lib-0.11.1 #159

Dependency resolution fails to install cyclonedx-python-lib-0.11.1 #159

Comments

tetsuo-cpp commented Dec 2, 2021 • edited Loading

Bug description

Reproduction steps

Expected behavior

Screenshots and logs

Platform information

tetsuo-cpp commented Dec 2, 2021 • edited Loading

woodruffw commented Dec 2, 2021 • edited Loading

woodruffw commented Dec 2, 2021

woodruffw commented Dec 2, 2021

woodruffw commented Dec 2, 2021

woodruffw commented Dec 2, 2021

woodruffw commented Dec 2, 2021

woodruffw commented Dec 2, 2021

di commented Dec 2, 2021

woodruffw commented Dec 2, 2021

hschilling commented Dec 4, 2021 • edited Loading

di commented Dec 4, 2021

hschilling commented Dec 4, 2021

Dependency resolution fails to install `cyclonedx-python-lib-0.11.1` #159

Dependency resolution fails to install `cyclonedx-python-lib-0.11.1` #159

tetsuo-cpp commented Dec 2, 2021 •

edited

Loading

tetsuo-cpp commented Dec 2, 2021 •

edited

Loading

woodruffw commented Dec 2, 2021 •

edited

Loading

hschilling commented Dec 4, 2021 •

edited

Loading