Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pipenv not honoring setup_requires while locking #4231

Closed
thehesiod opened this issue May 3, 2020 · 19 comments · Fixed by #4248
Closed

pipenv not honoring setup_requires while locking #4231

thehesiod opened this issue May 3, 2020 · 19 comments · Fixed by #4248
Labels
Category: Dependency Resolution Priority: Critical Status: In Progress Type: Bug 🐛 Type: Regression Type: Release Blocker

Comments

@thehesiod
Copy link

@thehesiod thehesiod commented May 3, 2020

Issue description

For setup.py's which have setup_requires like cython, pipenv does not seem to be honoring it while locking the Pipfile

Expected result

Should be able to lock with setup.py requirements

Actual result

alex@amohr-C02C14YVMD6M /tmp/foo $ cat Pipfile 
[[source]]
name = "pypi"
url = "https://pypi.fbn.org/simple"
verify_ssl = true

[packages]
ddtrace = {extras = ["profiling"], version = "==0.36.1" }

[requires]
python_version = "3.8"
alex@amohr-C02C14YVMD6M /tmp/foo $ pipenv lock --verbose
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
⠙ Locking...
✘ Locking Failed! 
                          ROUND 1                           
Current constraints:
  ddtrace[profiling]==0.36.1 (from -r /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/pipenvsa2yqfuyrequirements/pipenv-qgladdka-constraints.txt (line 2))

Finding the best candidates:
  found candidate ddtrace[profiling]==0.36.1 (constraint was ==0.36.1)

Finding secondary dependencies:
  ddtrace[profiling]==0.36.1 not in cache, need to check index
ERROR:pip.subprocessor:Command errored out with exit status 1:
 command: /Users/alex/.local/share/virtualenvs/foo-mzLZXOua/bin/python /usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/tmp7_75fjjs
     cwd: /private/var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/pip-resolver-uciotq_i/ddtrace
Complete output (14 lines):
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 257, in <module>
    main()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 240, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
    return hook(metadata_directory, config_settings)
  File "/Users/alex/.local/share/virtualenvs/foo-mzLZXOua/lib/python3.8/site-packages/setuptools/build_meta.py", line 158, in prepare_metadata_for_build_wheel
    self.run_setup()
  File "/Users/alex/.local/share/virtualenvs/foo-mzLZXOua/lib/python3.8/site-packages/setuptools/build_meta.py", line 143, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 9, in <module>
    from Cython.Build import cythonize  # noqa: I100
ModuleNotFoundError: No module named 'Cython'
----------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in <module>
    main()
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
    results, resolver = resolve(
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
    return resolve_deps(
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1371, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1086, in actually_resolve_deps
    resolver.resolve()
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 807, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 268, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 383, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 226, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 347, in get_legacy_dependencies
    results, ireq = self.resolve_reqs(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 303, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 339, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 287, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 564, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 549, in _generate_metadata
    return generate_metadata(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/build/metadata.py", line 36, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 167, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 245, in _call_hook
    self._subprocess_runner(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 271, in runner
    call_subprocess(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: /Users/alex/.local/share/virtualenvs/foo-mzLZXOua/bin/python /usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/tmp7_75fjjs Check the logs for full command output.
ROUND 1                           
Current constraints:
  ddtrace[profiling]==0.36.1 (from -r /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/pipenvsa2yqfuyrequirements/pipenv-qgladdka-constraints.txt (line 2))

Finding the best candidates:
  found candidate ddtrace[profiling]==0.36.1 (constraint was ==0.36.1)

Finding secondary dependencies:
  ddtrace[profiling]==0.36.1 not in cache, need to check index
ERROR:pip.subprocessor:Command errored out with exit status 1:
 command: /Users/alex/.local/share/virtualenvs/foo-mzLZXOua/bin/python /usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/tmp7_75fjjs
     cwd: /private/var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/pip-resolver-uciotq_i/ddtrace
Complete output (14 lines):
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 257, in <module>
    main()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 240, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
    return hook(metadata_directory, config_settings)
  File "/Users/alex/.local/share/virtualenvs/foo-mzLZXOua/lib/python3.8/site-packages/setuptools/build_meta.py", line 158, in prepare_metadata_for_build_wheel
    self.run_setup()
  File "/Users/alex/.local/share/virtualenvs/foo-mzLZXOua/lib/python3.8/site-packages/setuptools/build_meta.py", line 143, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 9, in <module>
    from Cython.Build import cythonize  # noqa: I100
ModuleNotFoundError: No module named 'Cython'
----------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in <module>
    main()
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
    results, resolver = resolve(
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
    return resolve_deps(
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1371, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1086, in actually_resolve_deps
    resolver.resolve()
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 807, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 268, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 383, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 226, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 347, in get_legacy_dependencies
    results, ireq = self.resolve_reqs(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 303, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 339, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 287, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 564, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 549, in _generate_metadata
    return generate_metadata(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/build/metadata.py", line 36, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 167, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 245, in _call_hook
    self._subprocess_runner(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 271, in runner
    call_subprocess(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: /Users/alex/.local/share/virtualenvs/foo-mzLZXOua/bin/python /usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/tmp7_75fjjs Check the logs for full command output.

Steps to replicate

See Pipfile above and lock command run


$ pipenv --support

Pipenv version: '2020.4.1b1'

Pipenv location: '/usr/local/lib/python3.8/site-packages/pipenv'

Python location: '/usr/local/opt/python@3.8/bin/python3.8'

Python installations found:

  • 3.8.2: /usr/local/opt/python@3.8/bin/python3
  • 3.8.2: /usr/local/opt/python@3.8/bin/python3.8
  • 3.8.0: /usr/local/bin/python3.8
  • 3.7.3: /usr/bin/python3
  • 3.7.0: /usr/local/bin/python3
  • 3.7.0: /usr/local/bin/python3.7m
  • 3.7.0: /usr/local/bin/python3.7
  • 3.6.7: /usr/local/bin/python3.6
  • 3.6.7: /usr/local/bin/python3.6m
  • 3.5.9: /Users/alex/.pyenv/versions/3.5.9/bin/python3
  • 2.7.16: /usr/bin/python2
  • 2.7.16: /usr/bin/python2.7
  • 2.7.15: /usr/local/bin/python2
  • 2.7.15: /usr/local/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.2',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '19.4.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 PST '
                     '2020; root:xnu-6153.101.6~15/RELEASE_X86_64',
 'python_full_version': '3.8.2',
 'python_version': '3.8',
 'sys_platform': 'darwin'}

System environment variables:

  • SHELL
  • XPC_FLAGS
  • HISTCONTROL
  • TERM_PROGRAM_VERSION
  • PKG_CONFIG_PATH
  • HISTSIZE
  • HDF5_DIR
  • SSH_AUTH_SOCK
  • TERM_SESSION_ID
  • GPG_TTY
  • serverFlavor
  • PWD
  • LOGNAME
  • MANPATH
  • LaunchInstanceID
  • HOME
  • LANG
  • SECURITYSESSIONID
  • TMPDIR
  • CLICOLOR
  • FBNSECRETS_FORCE_USER
  • OPENSC_LIBS
  • NVM_DIR
  • TERM
  • USER
  • DISPLAY
  • SHLVL
  • XPC_SERVICE_NAME
  • HOMEBREW_GITHUB_API_TOKEN
  • PS1
  • PATH
  • OLDPWD
  • GOPATH
  • TERM_PROGRAM
  • _
  • __CF_USER_TEXT_ENCODING
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONDONTWRITEBYTECODE
  • PIP_SHIMS_BASE_MODULE
  • PIP_PYTHON_PATH
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: ~/.local/bin:/usr/local/opt/python@3.8/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/Apple/usr/bin:/Users/alex/bin:/usr/local/opt/go/libexec/bin:/Users/alex/gowork/bin:/Applications/Postgres.app/Contents/Versions/9.6/bin:/usr/local/bin:/usr/local/sbin:/Users/alex/Downloads/google-cloud-sdk/bin:/Users/alex/dev/fbn.com/ops/docker/ecr/bin
  • SHELL: /usr/local/bin/bash
  • LANG: en_US.UTF-8
  • PWD: /tmp/foo

Contents of Pipfile ('/private/tmp/foo/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.fbn.org/simple"
verify_ssl = true

[packages]
ddtrace = {extras = ["profiling"], version = "==0.36.1" }

[requires]
python_version = "3.8"
@ncoghlan
Copy link
Member

@ncoghlan ncoghlan commented May 4, 2020

@techalchemy I think I hit a variant of this while testing 2020.4.1.b1, where attempting to lock failed on generating lazy-object-proxy metadata due to picking up an old setuptools from the existing virtual environment.

What's especially weird about my case of this is that I tried to check what happened if I ran python3 -m pipenv run pip wheel lazy-object-proxy directly, and that didn't try to build from the sdist at all, it used the published wheel, so there's an extra question as to why pep517 is event attempting to get metadata from the sdist when there's a valid wheel available.

[ncoghlan@thechalk]$ pipenv lock
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✘ Locking Failed! 
ERROR:pip.subprocessor:Command errored out with exit status 1:
 command: /home/ncoghlan/.local/share/virtualenvs/veefil-core-ztnLcQP0/bin/python /home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp4yv04ovl
     cwd: /tmp/pip-resolver-t8wle1jw/lazy-object-proxy
Complete output (10 lines):
Traceback (most recent call last):
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 257, in <module>
    main()
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 240, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 101, in prepare_metadata_for_build_wheel
    backend = _build_backend()
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 76, in _build_backend
    obj = getattr(obj, path_part)
AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'
----------------------------------------
Traceback (most recent call last):
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/resolver.py", line 807, in <module>
    main()
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/resolver.py", line 803, in main
    parsed.requirements_dir, parsed.packages, parse_only=parsed.parse_only)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/resolver.py", line 785, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/resolver.py", line 753, in resolve_packages
    requirements_dir=requirements_dir,
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/resolver.py", line 736, in resolve
    req_dir=requirements_dir
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/utils.py", line 1379, in resolve_deps
    req_dir=req_dir,
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/utils.py", line 1086, in actually_resolve_deps
    resolver.resolve()
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/utils.py", line 807, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 268, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 383, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 226, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 348, in get_legacy_dependencies
    download_dir, ireq, wheel_cache
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 303, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 339, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 287, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 509, in prepare_linked_requirement
    req, self.req_tracker, self.finder, self.build_isolation,
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 564, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 551, in _generate_metadata
    backend=self.pep517_backend,
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/operations/build/metadata.py", line 37, in generate_metadata
    metadata_dir
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 170, in prepare_metadata_for_build_wheel
    '_allow_fallback': _allow_fallback,
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 248, in _call_hook
    extra_environ=extra_environ
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 275, in runner
    spinner=spinner,
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: /home/ncoghlan/.local/share/virtualenvs/veefil-core-ztnLcQP0/bin/python /home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp4yv04ovl Check the logs for full command output.
[ncoghlan@thechalk]$ pipenv run pip install --upgrade setuptools
Collecting setuptools
  Using cached https://files.pythonhosted.org/packages/a0/df/635cdb901ee4a8a42ec68e480c49f85f4c59e8816effbf57d9e6ee8b3588/setuptools-46.1.3-py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 40.2.0
    Uninstalling setuptools-40.2.0:
      Successfully uninstalled setuptools-40.2.0
Successfully installed setuptools-46.1.3
[ncoghlan@thechalk]$ pipenv lock
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (393e59)!

@ncoghlan
Copy link
Member

@ncoghlan ncoghlan commented May 4, 2020

pypa/pip#6264 is a potentially related bug report against pip, where something goes wrong with the sys.path construction in the PEP 517 build environment. If that's what I'm hitting though, then it means I'm wrong about my failure having the same root cause as @thehesiod's failure.

@ncoghlan
Copy link
Member

@ncoghlan ncoghlan commented May 4, 2020

OK, looking at https://libraries.io/pypi/ddtrace and https://pypi.org/project/ddtrace/#files, I think the clearly common part of my error and @thehesiod's is that pipenv isn't using the pre-built wheel files on PyPI for metadata parsing.

The second part of the failure might be common: both projects have a pyproject.toml in their sdists, and while it's clear from @thehesiod's traceback that the cython dependency for ddtrace isn't being honoured, for lazy-object-proxy, the dependency that isn't being honoured is the one that pep517 itself injects to ensure that setuptools is at least up to the version that provides a legacy PEP 517 backend:

@thehesiod
Copy link
Author

@thehesiod thehesiod commented May 4, 2020

Btw this happens with the "current" version of pipenv as well

@miguelsanchez-eb
Copy link

@miguelsanchez-eb miguelsanchez-eb commented May 4, 2020

I'm having this issue as well.
I'm installing my dependencies system-wide inside a docker container using docker-compose:
docker-compose run my_container pipenv --dev --system --deploy install.

Then if I try to generate a lock file, it fails because it doesn't find Cython (I'm trying to install ddtrace as the OP).
docker-compose run my_container pipenv lock.

But if I generate the lock file in the host machine (my computer) it works fine.

Also, if I install manually Cython inside my docker machine and then try to run the pipenv lock, it works too.
I guess the problem could be pipenv lock creating a virtual env automatically and not being able to find Cython inside.

@thehesiod
Copy link
Author

@thehesiod thehesiod commented May 4, 2020

you know what, downgrading pip to 19.1 seems to work

@ncoghlan
Copy link
Member

@ncoghlan ncoghlan commented May 5, 2020

I ran into the problem with pip 20.0.2 and 20.1, so a bad interaction with more recent pip releases is plausible.

Did 19.1.1 work for you, or only 19.1 itself?

@techalchemy
Copy link
Member

@techalchemy techalchemy commented May 6, 2020

I think part of this bug is related to an issue with a patch we have on pip to modify candidate sort order which winds up sorting sdists ahead of wheels during resolution.

For example, I am able to install the package in the original issue if I address that problem.

techalchemy added a commit that referenced this issue May 6, 2020
- `ignore_compatibility` is meant to resolve hashes into the lockfile
  after resolution happens
- We still want compatible items to be the ones we actually tell pip to
  install
- Fixes #4231

Signed-off-by: Dan Ryan <dan.ryan@canonical.com>
@techalchemy techalchemy added Category: Dependency Resolution Priority: Critical Status: In Progress Type: Bug 🐛 Type: Regression labels May 6, 2020
@triage-new-issues triage-new-issues bot removed the triage label May 6, 2020
@techalchemy techalchemy added this to the April 2020 Release milestone May 6, 2020
@thehesiod
Copy link
Author

@thehesiod thehesiod commented May 6, 2020

what's odd is that after I downgraded pip now it always works, even after bumping pip up again, there is something odd with pip and having multiple python versions installed, ex: I have 3.8 installed via brew and the official 3.8 via installer. will take some time to track down what's going on with that

@thehesiod
Copy link
Author

@thehesiod thehesiod commented May 6, 2020

I'm wondering now if it can also be related to setuptools, sorry not sure it's going to be possible to figure out why my system started working now after I downgraded pip, I didn't pay attention to what other packages it downgraded when doing that. Btw just validated this fails with an empty Pipfile:
pipenv install ddtrace==0.36.1 and that patch does not seem to fix this particular issue

@techalchemy
Copy link
Member

@techalchemy techalchemy commented May 7, 2020

@thehesiod so neither the current master branch nor the proposed patch fixes this issue for you? I can't seem to make this break locally

@techalchemy
Copy link
Member

@techalchemy techalchemy commented May 7, 2020

I have 3.8 installed via brew and the official 3.8 via installer. will take some time to track down what's going on with that

I am just focusing on the fact that this is affecting MacOS. Is that true for everyone? Is it reproducible on linux?

@miguelsanchez-eb do you have a sample docker file that reproduces this issue by any chance?

@brainwane brainwane added the Type: Release Blocker label May 7, 2020
@techalchemy
Copy link
Member

@techalchemy techalchemy commented May 7, 2020

once again reading through this a bit more carefully I can see that this might only be reproducible if you have an environment that was created some time ago and has a copy of setuptools that is sufficiently outdated as to lack the fallback build backend.

I wonder if it's "good enough" to just listen for the specific failure mode (i.e. pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: .*/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel) and react by updating setuptools... I wonder if anyone is relying on an old version of setuptools for anything....

@techalchemy
Copy link
Member

@techalchemy techalchemy commented May 8, 2020

As my final note for the evening I am able to reproduce this by creating a virtualenv with an old version of setuptools (I initially tried <41.0 but that seemed to work ok, while the implicated version 40.2.0 seems to be completely broken for these packages). I have just wrapped up some changes that will implement the functionality I described above, i.e. upgrading setuptools if pipenv bumps into this bug and re-resolving.

After looking over my own libraries I've realized that where I handle resolution of packages with no backend specified and no build requirements, I begin by forcing setuptools>=40.8 so I assume this is the version where the __legacy__ backend was added.

Tomorrow morning I will throw together a fake package with a setuptools.build_meta:__legacy__ backend and a quick test. If anyone foresees any issues with this approach, please raise them

@thehesiod
Copy link
Author

@thehesiod thehesiod commented May 8, 2020

thank you for all the time researching this!

@thehesiod
Copy link
Author

@thehesiod thehesiod commented May 8, 2020

perhaps pipenv needs to ensure you have an appropriately new setuptools? it would be nice if python did this at launch itself by checking the package requirements :(

@ncoghlan
Copy link
Member

@ncoghlan ncoghlan commented May 9, 2020

pip tries to ensure that setuptools is new enough when running PEP 517/518 builds, so I'm thinking there may be some subtlety that is bypassing that logic in the metadata retrieval case.

techalchemy added a commit that referenced this issue May 9, 2020
- `ignore_compatibility` is meant to resolve hashes into the lockfile
  after resolution happens
- We still want compatible items to be the ones we actually tell pip to
  install
- Fixes #4231

Signed-off-by: Dan Ryan <dan.ryan@canonical.com>
techalchemy added a commit that referenced this issue May 9, 2020
- Fix how `use_pep517` and `build_isolation` are read from the
  environment -- introduce a new environment helper to detect
  `<PREFIX>_<SETTING>` and `<PREFIX>_NO_<SETTING>`, check for booleans
  and return appropriately boolean, str, or None types
- Check for `False` values when adding `--no-use-pep517` and
  `--no-build-isolation` during resolution rather than falsey values
- Change environment variable name from `PIP_PYTHON_VERSION` to
  `PIPENV_REQUESTED_PYTHON_VERSION` to avoid causing `pip` to fail due
  to accidentally percieving the `python_version` flag as being set --
  this is an artifact from attempting to resolve outside of the
  virtualenv
- Add `pipenv` to the path of patched `notpip.__main__` to accommodate
  updated import fully qualified module names
- Update `pip` and `piptools` patches
- Add test packages for each of two known failure modes: outdated
  `setuptools` with a missing `build-backend` (which `pip` forces to
  `build_meta:__legacy__` & which doesn't exist before `40.8`), and
  `import Cython` statements in `setup.py` in packages with properly
  defined `pyproject.toml` `build-backend` lines.
- Fixes #4231
- Replaces, includes, and closes #4242

Signed-off-by: Dan Ryan <dan.ryan@canonical.com>

Add integration tests for #4231

Signed-off-by: Dan Ryan <dan.ryan@canonical.com>
techalchemy added a commit that referenced this issue May 9, 2020
- Fix how `use_pep517` and `build_isolation` are read from the
  environment -- introduce a new environment helper to detect
  `<PREFIX>_<SETTING>` and `<PREFIX>_NO_<SETTING>`, check for booleans
  and return appropriately boolean, str, or None types
- Check for `False` values when adding `--no-use-pep517` and
  `--no-build-isolation` during resolution rather than falsey values
- Change environment variable name from `PIP_PYTHON_VERSION` to
  `PIPENV_REQUESTED_PYTHON_VERSION` to avoid causing `pip` to fail due
  to accidentally percieving the `python_version` flag as being set --
  this is an artifact from attempting to resolve outside of the
  virtualenv
- Add `pipenv` to the path of patched `notpip.__main__` to accommodate
  updated import fully qualified module names
- Update `pip` and `piptools` patches
- Add test packages for each of two known failure modes: outdated
  `setuptools` with a missing `build-backend` (which `pip` forces to
  `build_meta:__legacy__` & which doesn't exist before `40.8`), and
  `import Cython` statements in `setup.py` in packages with properly
  defined `pyproject.toml` `build-backend` lines.
- Fixes #4231
- Replaces, includes, and closes #4242

Signed-off-by: Dan Ryan <dan.ryan@canonical.com>

Add integration tests for #4231

Signed-off-by: Dan Ryan <dan.ryan@canonical.com>
@techalchemy
Copy link
Member

@techalchemy techalchemy commented May 9, 2020

Hey so I have a more thorough grasp of this issue now and there are actually two separate issues in play which led me down a few confusing paths before sorting this out.

Firstly, the issue with lazy-object-proxy -- the package in question has a setup.py which takes advantage of setuptools-scm (other issues may come up with regard to this because it probably doesn't work properly when resolving sdists either), as well as having a pyproject.toml with build-system.requires defined, but no build-system.build-backend.

In our resolution step, pip was attempting to set the build-system.build-backend to setuptools.build_meta:__legacy__ which, if you've had your virtualenv around and never updated setuptools, might not exist. This normally doesn't matter, since pip uses its own internal pep517 build backend and builds in an isolate build environment.

In the case of ddtrace, there is also a pyproject.toml file, which, in this case, includes a sufficiently unconstrained build-system.requires and explicitly sets build-system.build-backend to setuptools.build_meta. In setup.py, however, it imports Cython, one of its build-system.requires items. In another issue I speculated that we might fix our dependency resolution challenges if we enabled build isolation during dependency resolution.

Following up on that, I discovered that when we are determining whether to build in isolation or use pep517, we are doing this a bit...wrong.

In this first bit of resolver code, you can see that we were relying on truthy values of use_pep517 and build_isolation rather than the more strong is False comparison -- i.e., if is_pep517 is None, we would still decide to pass --no-use-pep517 to pip.

-    def prepare_pip_args(self, use_pep517=True, build_isolation=True):
+    def prepare_pip_args(self, use_pep517=False, build_isolation=True):
         pip_args = []
         if self.sources:
             pip_args = prepare_pip_source_args(self.sources, pip_args)
-        if not use_pep517:
+        if use_pep517 is False:
             pip_args.append("--no-use-pep517")
-        if not build_isolation:
+        if build_isolation is False:
             pip_args.append("--no-build-isolation")
         pip_args.extend(["--cache-dir", environments.PIPENV_CACHE_DIR])
         return pip_args

And here where we decide what actually gets passed in, we were doing some overly complicated logical expressions which wind up evaluating in an undesirable way. To break the issue down: use_pep517 = False if (condition1) else (True if condition2 else None). So, even if this accurately evaluated to None, we still read that as --no-<x>.

     @property
     def pip_args(self):
-        use_pep517 = False if (
-            os.environ.get("PIP_NO_USE_PEP517", None) is not None
-        ) else (True if os.environ.get("PIP_USE_PEP517", None) is not None else None)
-        build_isolation = False if (
-            os.environ.get("PIP_NO_BUILD_ISOLATION", None) is not None
-        ) else (True if os.environ.get("PIP_BUILD_ISOLATION", None) is not None else None)
+        use_pep517 = environments.get_from_env("USE_PEP517", prefix="PIP")
+        build_isolation = environments.get_from_env("BUILD_ISOLATION", prefix="PIP")
         if self._pip_args is None:
             self._pip_args = self.prepare_pip_args(
                 use_pep517=use_pep517, build_isolation=build_isolation

As a result I've provided a more general function for reading these types of values from the environment in the linked PR and a way to build in isolation by default. I'm a little bit unsure about whether this might break certain workflows so I want to consider whether we need any additional fallbacks for those cases.

That said, thanks for providing the detail required to troubleshoot this one, it was definitely tricky!

@thehesiod
Copy link
Author

@thehesiod thehesiod commented May 9, 2020

OK, looking at https://libraries.io/pypi/ddtrace and https://pypi.org/project/ddtrace/#files, I think the clearly common part of my error and @thehesiod's is that pipenv isn't using the pre-built wheel files on PyPI for metadata parsing.

btw not using the pre-built wheel is probably due to our own pypi server, I forgot to upload the wheels, and we only had the sdist with the pypi server setting to not forward missing files to pypi.org for security/custom env purposes.

fwojciak pushed a commit to fwojciak/pipenv that referenced this issue May 29, 2020
2020.5.28 (2020-05-28)
======================

Features & Improvements
-----------------------

-   `pipenv install` and `pipenv sync` will no longer attempt to install satisfied dependencies during installation. pypa#3057, pypa#3506
-   Added support for resolution of direct-url dependencies in `setup.py` files to respect `PEP-508` style URL dependencies. pypa#3148
-   Added full support for resolution of all dependency types including direct URLs, zip archives, tarballs, etc.
    -   Improved error handling and formatting.
    -   Introduced improved cross platform stream wrappers for better `stdout` and `stderr` consistency. pypa#3298
-   For consistency with other commands and the `--dev` option description, `pipenv lock --requirements --dev` now emits both default and development dependencies. The new `--dev-only` option requests the previous behaviour (e.g. to generate a `dev-requirements.txt` file). pypa#3316
-   Pipenv will now successfully recursively lock VCS sub-dependencies. pypa#3328
-   Added support for `--verbose` output to `pipenv run`. pypa#3348
-   Pipenv will now discover and resolve the intrinsic dependencies of **all** VCS dependencies, whether they are editable or not, to prevent resolution conflicts. pypa#3368
-   Added a new environment variable, `PIPENV_RESOLVE_VCS`, to toggle dependency resolution off for non-editable VCS, file, and URL based dependencies. pypa#3577
-   Added the ability for Windows users to enable emojis by setting `PIPENV_HIDE_EMOJIS=0`. pypa#3595
-   Allow overriding `PIPENV_INSTALL_TIMEOUT` environment variable (in seconds). pypa#3652
-   Allow overriding `PIP_EXISTS_ACTION` evironment variable (value is passed to pip install). Possible values here: <https://pip.pypa.io/en/stable/reference/pip/#exists-action-option> Useful when you need to `PIP\_EXISTS\_ACTION=i` (ignore existing packages) - great for CI environments, where you need really fast setup. pypa#3738
-   Pipenv will no longer forcibly override `PIP_NO_DEPS` on all vcs and file dependencies as resolution happens on these in a pre-lock step. pypa#3763
-   Improved verbose logging output during `pipenv lock` will now stream output to the console while maintaining a spinner. pypa#3810
-   Added support for automatic python installs via `asdf` and associated `PIPENV_DONT_USE_ASDF` environment variable. pypa#4018
-   Pyenv/asdf can now be used whether or not they are available on PATH. Setting `PYENV_ROOT`/`ASDF_DIR` in a `.env` file allows Pipenv to install an interpreter without any shell customizations, so long as pyenv/asdf is installed. pypa#4245
-   Added `--key` command line parameter for including personal PyUp.io API tokens when running `pipenv check`. pypa#4257

Behavior Changes
----------------

-   Make conservative checks of known exceptions when subprocess returns output, so user won\'t see the whole traceback - just the error. pypa#2553
-   Do not touch Pipfile early and rely on it so that one can do `pipenv sync` without a Pipfile. pypa#3386
-   Re-enable `--help` option for `pipenv run` command. pypa#3844
-   Make sure `pipenv lock -r --pypi-mirror {MIRROR_URL}` will respect the pypi-mirror in requirements output. pypa#4199

Bug Fixes
---------

-   Raise `PipenvUsageError` when \[\[source\]\] does not contain url field. pypa#2373
-   Fixed a bug which caused editable package resolution to sometimes fail with an unhelpful setuptools-related error message. pypa#2722
-   Fixed an issue which caused errors due to reliance on the system utilities `which` and `where` which may not always exist on some
    systems.
-   Fixed a bug which caused periodic failures in python discovery when executables named `python` were not present on the target `$PATH`. pypa#2783
-   Dependency resolution now writes hashes for local and remote files to the lockfile. pypa#3053
-   Fixed a bug which prevented `pipenv graph` from correctly showing all dependencies when running from within `pipenv shell`. pypa#3071
-   Fixed resolution of direct-url dependencies in `setup.py` files to respect `PEP-508` style URL dependencies. pypa#3148
-   Fixed a bug which caused failures in warning reporting when running pipenv inside a virtualenv under some circumstances.
-   Fixed a bug with package discovery when running `pipenv clean`. pypa#3298
-   Quote command arguments with carets (`^`) on Windows to work around unintended shell escapes. pypa#3307
-   Handle alternate names for UTF-8 encoding. pypa#3313
-   Abort pipenv before adding the non-exist package to Pipfile. pypa#3318
-   Don\'t normalize the package name user passes in. pypa#3324
-   Fix a bug where custom virtualenv can not be activated with pipenv shell pypa#3339
-   Fix a bug that `--site-packages` flag is not recognized. pypa#3351
-   Fix a bug where `pipenv --clear` is not working pypa#3353
-   Fix unhashable type error during `$ pipenv install --selective-upgrade` pypa#3384
-   Dependencies with direct `PEP508` compliant VCS URLs specified in their `install_requires` will now be successfully locked during the resolution process. pypa#3396
-   Fixed a keyerror which could occur when locking VCS dependencies in
    some cases. pypa#3404
-   Fixed a bug that `ValidationError` is thrown when some fields are missing in source section. pypa#3427
-   Updated the index names in lock file when source name in Pipfile is changed. pypa#3449
-   Fixed an issue which caused `pipenv install --help` to show duplicate entries for `--pre`. pypa#3479
-   Fix bug causing `[SSL: CERTIFICATE_VERIFY_FAILED]` when Pipfile `[[source]]` has `verify_ssl=false` and url with custom port. pypa#3502
-   Fix `sync --sequential` ignoring `pip install` errors and logs. pypa#3537
-   Fix the issue that lock file can\'t be created when `PIPENV_PIPFILE` is not under working directory. pypa#3584
-   Pipenv will no longer inadvertently set `editable=True` on all vcs dependencies. pypa#3647
-   The `--keep-outdated` argument to `pipenv install` and `pipenv lock` will now drop specifier constraints when encountering editable dependencies.
    -   In addition, `--keep-outdated` will retain specifiers that would otherwise be dropped from any entries that have not been updated. pypa#3656
-   Fixed a bug which sometimes caused pipenv to fail to respect the `--site-packages` flag when passed with `pipenv install`. pypa#3718
-   Normalize the package names to lowercase when comparing used and in-Pipfile packages. pypa#3745
-   `pipenv update --outdated` will now correctly handle comparisons between pre/post-releases and normal releases. pypa#3766
-   Fixed a `KeyError` which could occur when pinning outdated VCS dependencies via `pipenv lock --keep-outdated`. pypa#3768
-   Resolved an issue which caused resolution to fail when encountering poorly formatted `python_version` markers in `setup.py` and `setup.cfg` files. pypa#3786
-   Fix a bug that installation errors are displayed as a list. pypa#3794
-   Update `pythonfinder` to fix a problem that `python.exe` will be mistakenly chosen for virtualenv creation under WSL. pypa#3807
-   Fixed several bugs which could prevent editable VCS dependencies from being installed into target environments, even when reporting
    successful installation. pypa#3809
-   `pipenv check --system` should find the correct Python interpreter when `python` does not exist on the system. pypa#3819
-   Resolve the symlinks when the path is absolute. pypa#3842
-   Pass `--pre` and `--clear` options to `pipenv update --outdated`. pypa#3879
-   Fixed a bug which prevented resolution of direct URL dependencies which have PEP508 style direct url VCS sub-dependencies with
    subdirectories. pypa#3976
-   Honor `PIPENV_SPINNER` environment variable pypa#4045
-   Fixed an issue with `pipenv check` failing due to an invalid API key from `pyup.io`. pypa#4188
-   Fixed a bug which caused versions from VCS dependencies to be included in `Pipfile.lock` inadvertently. pypa#4217
-   Fixed a bug which caused pipenv to search non-existent virtual environments for `pip` when installing using `--system`. pypa#4220
-   `Requires-Python` values specifying constraint versions of python starting from `1.x` will now be parsed successfully. pypa#4226
-   Fix a bug of `pipenv update --outdated` that can\'t print output correctly. pypa#4229
-   Fixed a bug which caused pipenv to prefer source distributions over wheels from `PyPI` during the dependency resolution phase. Fixed an issue which prevented proper build isolation using `pep517` based builders during dependency resolution. pypa#4231
-   Don\'t fallback to system Python when no matching Python version is found. pypa#4232

Vendored Libraries
------------------

- Updated `pip_shims` to support `--outdated` with new pip versions. pypa#3766
- Update vendored dependencies and invocations
  - Update vendored and patched dependencies
  - Update patches on `piptools`, `pip`, `pip-shims`, `tomlkit`
  - Fix invocations of dependencies
  - Fix custom `InstallCommand` instantiation
  - Update `PackageFinder` usage
  - Fix `Bool` stringify attempts from `tomlkit`
  - Updated vendored dependencies:
    -   **attrs**: `18.2.0 => `19.1.0`
    -   **certifi**: `2018.10.15 => `2019.3.9`
    -   **cached\_property**: `1.4.3 => `1.5.1`
    -   **cerberus**: `1.2.0 => `1.3.1`
    -   **click**: `7.0.0 => `7.1.1`
    -   **click-completion**: `0.5.0 => `0.5.1`
    -   **colorama**: `0.3.9 => `0.4.3`
    -   **contextlib2**: `(new) => `0.6.0.post1`
    -   **distlib**: `0.2.8 => `0.2.9`
    -   **funcsigs**: `(new) => `1.0.2`
    -   **importlib\_metadata** `1.3.0 => `1.5.1`
    -   **importlib-resources**: `(new) => `1.4.0`
    -   **idna**: `2.7 => `2.9`
    -   **jinja2**: `2.10.0 => `2.11.1`
    -   **markupsafe**: `1.0 => `1.1.1`
    -   **more-itertools**: `(new) => `5.0.0`
    -   **orderedmultidict**: `(new) => `1.0`
    -   **packaging**: `18.0 => `19.0`
    -   **parse**: `1.9.0 => `1.15.0`
    -   **pathlib2**: `2.3.2 => `2.3.3`
    -   **pep517**: `(new) => `0.5.0`
    -   **pexpect**: `4.6.0 => `4.8.0`
    -   **pip-shims**: `0.2.0 => `0.5.1`
    -   **pipdeptree**: `0.13.0 => `0.13.2`
    -   **pyparsing**: `2.2.2 => `2.4.6`
    -   **python-dotenv**: `0.9.1 => `0.10.2`
    -   **pythonfinder**: `1.1.10 => `1.2.2`
    -   **pytoml**: `(new) => `0.1.20`
    -   **requests**: `2.20.1 => `2.23.0`
    -   **requirementslib**: `1.3.3 => `1.5.4`
    -   **scandir**: `1.9.0 => `1.10.0`
    -   **shellingham**: `1.2.7 => `1.3.2`
    -   **six**: `1.11.0 => `1.14.0`
    -   **tomlkit**: `0.5.2 => `0.5.11`
    -   **urllib3**: `1.24 => `1.25.8`
    -   **vistir**: `0.3.0 => `0.5.0`
    -   **yaspin**: `0.14.0 => `0.14.3`
    -   **zipp**: `0.6.0`
    - Removed vendored dependency **cursor**. pypa#4169

-   Add and update vendored dependencies to accommodate `safety` vendoring:
    -   **safety** `(none)` => `1.8.7`
    -   **dparse** `(none)` => `0.5.0`
    -   **pyyaml** `(none)` => `5.3.1`
    -   **urllib3** `1.25.8` => `1.25.9`
    -   **certifi** `2019.11.28` => `2020.4.5.1`
    -   **pyparsing** `2.4.6` => `2.4.7`
    -   **resolvelib** `0.2.2` => `0.3.0`
    -   **importlib-metadata** `1.5.1` => `1.6.0`
    -   **pip-shims** `0.5.1` => `0.5.2`
    -   **requirementslib** `1.5.5` => `1.5.6` pypa#4188

-   Updated vendored `pip` => `20.0.2` and `pip-tools` => `5.0.0`. pypa#4215
-   Updated vendored dependencies to latest versions for security and bug fixes:
    -   **requirementslib** `1.5.8` => `1.5.9`
    -   **vistir** `0.5.0` => `0.5.1`
    -   **jinja2** `2.11.1` => `2.11.2`
    -   **click** `7.1.1` => `7.1.2`
    -   **dateutil** `(none)` => `2.8.1`
    -   **backports.functools\_lru\_cache** `1.5.0` => `1.6.1`
    -   **enum34** `1.1.6` => `1.1.10`
    -   **toml** `0.10.0` => `0.10.1`
    -   **importlib\_resources** `1.4.0` => `1.5.0` pypa#4226
-   Changed attrs import path in vendored dependencies to always import from `pipenv.vendor`. pypa#4267

Improved Documentation
----------------------

-   Added documenation about variable expansion in `Pipfile` entries. pypa#2317
-   Consolidate all contributing docs in the rst file pypa#3120
-   Update the out-dated manual page. pypa#3246
-   Move CLI docs to its own page. pypa#3346
-   Replace (non-existant) video on docs index.rst with equivalent gif. pypa#3499
-   Clarify wording in Basic Usage example on using double quotes to escape shell redirection pypa#3522
-   Ensure docs show navigation on small-screen devices pypa#3527
-   Added a link to the TOML Spec under General Recommendations & Version Control to clarify how Pipfiles should be written. pypa#3629
-   Updated the documentation with the new `pytest` entrypoint. pypa#3759
-   Fix link to GIF in README.md demonstrating Pipenv\'s usage, and add descriptive alt text. pypa#3911
-   Added a line describing potential issues in fancy extension. pypa#3912
-   Documental description of how Pipfile works and association with Pipenv. pypa#3913
-   Clarify the proper value of `python_version` and `python_full_version`. pypa#3914
-   Write description for `--deploy` extension and few extensions differences. pypa#3915
-   More documentation for `.env` files pypa#4100
-   Updated documentation to point to working links. pypa#4137
-   Replace docs.pipenv.org with pipenv.pypa.io pypa#4167
-   Added functionality to check spelling in documentation and cleaned up existing typographical issues. pypa#4209
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category: Dependency Resolution Priority: Critical Status: In Progress Type: Bug 🐛 Type: Regression Type: Release Blocker
Projects
None yet
5 participants