pipenv not honoring setup_requires while locking

[thehesiod]

Issue description

For setup.py's which have setup_requires like cython, pipenv does not seem to be honoring it while locking the Pipfile

Expected result

Should be able to lock with setup.py requirements

Actual result

alex@amohr-C02C14YVMD6M /tmp/foo $ cat Pipfile 
[[source]]
name = "pypi"
url = "https://pypi.fbn.org/simple"
verify_ssl = true

[packages]
ddtrace = {extras = ["profiling"], version = "==0.36.1" }

[requires]
python_version = "3.8"
alex@amohr-C02C14YVMD6M /tmp/foo $ pipenv lock --verbose
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
⠙ Locking...
✘ Locking Failed! 
                          ROUND 1                           
Current constraints:
  ddtrace[profiling]==0.36.1 (from -r /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/pipenvsa2yqfuyrequirements/pipenv-qgladdka-constraints.txt (line 2))

Finding the best candidates:
  found candidate ddtrace[profiling]==0.36.1 (constraint was ==0.36.1)

Finding secondary dependencies:
  ddtrace[profiling]==0.36.1 not in cache, need to check index
ERROR:pip.subprocessor:Command errored out with exit status 1:
 command: /Users/alex/.local/share/virtualenvs/foo-mzLZXOua/bin/python /usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/tmp7_75fjjs
     cwd: /private/var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/pip-resolver-uciotq_i/ddtrace
Complete output (14 lines):
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 257, in <module>
    main()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 240, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
    return hook(metadata_directory, config_settings)
  File "/Users/alex/.local/share/virtualenvs/foo-mzLZXOua/lib/python3.8/site-packages/setuptools/build_meta.py", line 158, in prepare_metadata_for_build_wheel
    self.run_setup()
  File "/Users/alex/.local/share/virtualenvs/foo-mzLZXOua/lib/python3.8/site-packages/setuptools/build_meta.py", line 143, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 9, in <module>
    from Cython.Build import cythonize  # noqa: I100
ModuleNotFoundError: No module named 'Cython'
----------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in <module>
    main()
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
    results, resolver = resolve(
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
    return resolve_deps(
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1371, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1086, in actually_resolve_deps
    resolver.resolve()
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 807, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 268, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 383, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 226, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 347, in get_legacy_dependencies
    results, ireq = self.resolve_reqs(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 303, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 339, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 287, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 564, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 549, in _generate_metadata
    return generate_metadata(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/build/metadata.py", line 36, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 167, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 245, in _call_hook
    self._subprocess_runner(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 271, in runner
    call_subprocess(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: /Users/alex/.local/share/virtualenvs/foo-mzLZXOua/bin/python /usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/tmp7_75fjjs Check the logs for full command output.
ROUND 1                           
Current constraints:
  ddtrace[profiling]==0.36.1 (from -r /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/pipenvsa2yqfuyrequirements/pipenv-qgladdka-constraints.txt (line 2))

Finding the best candidates:
  found candidate ddtrace[profiling]==0.36.1 (constraint was ==0.36.1)

Finding secondary dependencies:
  ddtrace[profiling]==0.36.1 not in cache, need to check index
ERROR:pip.subprocessor:Command errored out with exit status 1:
 command: /Users/alex/.local/share/virtualenvs/foo-mzLZXOua/bin/python /usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/tmp7_75fjjs
     cwd: /private/var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/pip-resolver-uciotq_i/ddtrace
Complete output (14 lines):
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 257, in <module>
    main()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 240, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
    return hook(metadata_directory, config_settings)
  File "/Users/alex/.local/share/virtualenvs/foo-mzLZXOua/lib/python3.8/site-packages/setuptools/build_meta.py", line 158, in prepare_metadata_for_build_wheel
    self.run_setup()
  File "/Users/alex/.local/share/virtualenvs/foo-mzLZXOua/lib/python3.8/site-packages/setuptools/build_meta.py", line 143, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 9, in <module>
    from Cython.Build import cythonize  # noqa: I100
ModuleNotFoundError: No module named 'Cython'
----------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in <module>
    main()
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
    results, resolver = resolve(
  File "/usr/local/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
    return resolve_deps(
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1371, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 1086, in actually_resolve_deps
    resolver.resolve()
  File "/usr/local/lib/python3.8/site-packages/pipenv/utils.py", line 807, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 268, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 383, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 226, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 347, in get_legacy_dependencies
    results, ireq = self.resolve_reqs(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 303, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 339, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 287, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 564, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 549, in _generate_metadata
    return generate_metadata(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/operations/build/metadata.py", line 36, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 167, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 245, in _call_hook
    self._subprocess_runner(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 271, in runner
    call_subprocess(
  File "/usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: /Users/alex/.local/share/virtualenvs/foo-mzLZXOua/bin/python /usr/local/lib/python3.8/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/my/1cvd396x2m35py37s4lfhjv80000gp/T/tmp7_75fjjs Check the logs for full command output.

Steps to replicate

See Pipfile above and lock command run

---

$ pipenv --support

Pipenv version: '2020.4.1b1'

Pipenv location: '/usr/local/lib/python3.8/site-packages/pipenv'

Python location: '/usr/local/opt/python@3.8/bin/python3.8'

Python installations found:

• 3.8.2: /usr/local/opt/python@3.8/bin/python3
• 3.8.2: /usr/local/opt/python@3.8/bin/python3.8
• 3.8.0: /usr/local/bin/python3.8
• 3.7.3: /usr/bin/python3
• 3.7.0: /usr/local/bin/python3
• 3.7.0: /usr/local/bin/python3.7m
• 3.7.0: /usr/local/bin/python3.7
• 3.6.7: /usr/local/bin/python3.6
• 3.6.7: /usr/local/bin/python3.6m
• 3.5.9: /Users/alex/.pyenv/versions/3.5.9/bin/python3
• 2.7.16: /usr/bin/python2
• 2.7.16: /usr/bin/python2.7
• 2.7.15: /usr/local/bin/python2
• 2.7.15: /usr/local/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.2',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '19.4.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 PST '
                     '2020; root:xnu-6153.101.6~15/RELEASE_X86_64',
 'python_full_version': '3.8.2',
 'python_version': '3.8',
 'sys_platform': 'darwin'}

System environment variables:

• SHELL
• XPC_FLAGS
• HISTCONTROL
• TERM_PROGRAM_VERSION
• PKG_CONFIG_PATH
• HISTSIZE
• HDF5_DIR
• SSH_AUTH_SOCK
• TERM_SESSION_ID
• GPG_TTY
• serverFlavor
• PWD
• LOGNAME
• MANPATH
• LaunchInstanceID
• HOME
• LANG
• SECURITYSESSIONID
• TMPDIR
• CLICOLOR
• FBNSECRETS_FORCE_USER
• OPENSC_LIBS
• NVM_DIR
• TERM
• USER
• DISPLAY
• SHLVL
• XPC_SERVICE_NAME
• HOMEBREW_GITHUB_API_TOKEN
• PS1
• PATH
• OLDPWD
• GOPATH
• TERM_PROGRAM
• _
• __CF_USER_TEXT_ENCODING
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: ~/.local/bin:/usr/local/opt/python@3.8/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/Apple/usr/bin:/Users/alex/bin:/usr/local/opt/go/libexec/bin:/Users/alex/gowork/bin:/Applications/Postgres.app/Contents/Versions/9.6/bin:/usr/local/bin:/usr/local/sbin:/Users/alex/Downloads/google-cloud-sdk/bin:/Users/alex/dev/fbn.com/ops/docker/ecr/bin
• SHELL: /usr/local/bin/bash
• LANG: en_US.UTF-8
• PWD: /tmp/foo

---

Contents of Pipfile ('/private/tmp/foo/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.fbn.org/simple"
verify_ssl = true

[packages]
ddtrace = {extras = ["profiling"], version = "==0.36.1" }

[requires]
python_version = "3.8"

[ncoghlan]

@techalchemy I think I hit a variant of this while testing 2020.4.1.b1, where attempting to lock failed on generating lazy-object-proxy metadata due to picking up an old setuptools from the existing virtual environment.

What's especially weird about my case of this is that I tried to check what happened if I ran python3 -m pipenv run pip wheel lazy-object-proxy directly, and that didn't try to build from the sdist at all, it used the published wheel, so there's an extra question as to why pep517 is event attempting to get metadata from the sdist when there's a valid wheel available.

[ncoghlan@thechalk]$ pipenv lock
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✘ Locking Failed! 
ERROR:pip.subprocessor:Command errored out with exit status 1:
 command: /home/ncoghlan/.local/share/virtualenvs/veefil-core-ztnLcQP0/bin/python /home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp4yv04ovl
     cwd: /tmp/pip-resolver-t8wle1jw/lazy-object-proxy
Complete output (10 lines):
Traceback (most recent call last):
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 257, in <module>
    main()
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 240, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 101, in prepare_metadata_for_build_wheel
    backend = _build_backend()
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 76, in _build_backend
    obj = getattr(obj, path_part)
AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'
----------------------------------------
Traceback (most recent call last):
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/resolver.py", line 807, in <module>
    main()
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/resolver.py", line 803, in main
    parsed.requirements_dir, parsed.packages, parse_only=parsed.parse_only)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/resolver.py", line 785, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/resolver.py", line 753, in resolve_packages
    requirements_dir=requirements_dir,
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/resolver.py", line 736, in resolve
    req_dir=requirements_dir
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/utils.py", line 1379, in resolve_deps
    req_dir=req_dir,
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/utils.py", line 1086, in actually_resolve_deps
    resolver.resolve()
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/utils.py", line 807, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 268, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 383, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 226, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 348, in get_legacy_dependencies
    download_dir, ireq, wheel_cache
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 303, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 339, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 287, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 509, in prepare_linked_requirement
    req, self.req_tracker, self.finder, self.build_isolation,
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 564, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 551, in _generate_metadata
    backend=self.pep517_backend,
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/operations/build/metadata.py", line 37, in generate_metadata
    metadata_dir
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 170, in prepare_metadata_for_build_wheel
    '_allow_fallback': _allow_fallback,
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 248, in _call_hook
    extra_environ=extra_environ
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 275, in runner
    spinner=spinner,
  File "/home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: /home/ncoghlan/.local/share/virtualenvs/veefil-core-ztnLcQP0/bin/python /home/ncoghlan/.local/lib/python3.7/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmp4yv04ovl Check the logs for full command output.
[ncoghlan@thechalk]$ pipenv run pip install --upgrade setuptools
Collecting setuptools
  Using cached https://files.pythonhosted.org/packages/a0/df/635cdb901ee4a8a42ec68e480c49f85f4c59e8816effbf57d9e6ee8b3588/setuptools-46.1.3-py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 40.2.0
    Uninstalling setuptools-40.2.0:
      Successfully uninstalled setuptools-40.2.0
Successfully installed setuptools-46.1.3
[ncoghlan@thechalk]$ pipenv lock
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (393e59)!

[ncoghlan]

pypa/pip#6264 is a potentially related bug report against pip, where something goes wrong with the sys.path construction in the PEP 517 build environment. If that's what I'm hitting though, then it means I'm wrong about my failure having the same root cause as @thehesiod's failure.

[ncoghlan]

OK, looking at https://libraries.io/pypi/ddtrace and https://pypi.org/project/ddtrace/#files, I think the clearly common part of my error and @thehesiod's is that pipenv isn't using the pre-built wheel files on PyPI for metadata parsing.

The second part of the failure might be common: both projects have a pyproject.toml in their sdists, and while it's clear from @thehesiod's traceback that the cython dependency for ddtrace isn't being honoured, for lazy-object-proxy, the dependency that isn't being honoured is the one that pep517 itself injects to ensure that setuptools is at least up to the version that provides a legacy PEP 517 backend:

pipenv/pipenv/patched/notpip/_internal/pyproject.py

Line 180 in 7e08bb9

if backend is None:

[thehesiod]

Btw this happens with the "current" version of pipenv as well

[miguelsanchez-eb]

I'm having this issue as well.
I'm installing my dependencies system-wide inside a docker container using docker-compose:
docker-compose run my_container pipenv --dev --system --deploy install.

Then if I try to generate a lock file, it fails because it doesn't find Cython (I'm trying to install ddtrace as the OP).
docker-compose run my_container pipenv lock.

But if I generate the lock file in the host machine (my computer) it works fine.

Also, if I install manually Cython inside my docker machine and then try to run the pipenv lock, it works too.
I guess the problem could be pipenv lock creating a virtual env automatically and not being able to find Cython inside.

[thehesiod]

you know what, downgrading pip to 19.1 seems to work

[ncoghlan]

I ran into the problem with pip 20.0.2 and 20.1, so a bad interaction with more recent pip releases is plausible.

Did 19.1.1 work for you, or only 19.1 itself?

[techalchemy]

I think part of this bug is related to an issue with a patch we have on pip to modify candidate sort order which winds up sorting sdists ahead of wheels during resolution.

For example, I am able to install the package in the original issue if I address that problem.