2021.11.5.post0, ubuntu 20.04: pipenv broken

[mjpieters]

On Ubuntu the pip version is.. special. In a way that breaks the new pipenv release:

$ pipenv lock
/usr/share/python-wheels/requests-2.22.0-py2.py3-none-any.whl/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.6) or chardet (3.0.4) doesn't match a supported version!
Creating a Pipfile for this project...
Traceback (most recent call last):
  File "/usr/local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/cli/command.py", line 322, in lock
    ensure_project(
  File "/usr/local/lib/python3.9/dist-packages/pipenv/core.py", line 560, in ensure_project
    ensure_pipfile(
  File "/usr/local/lib/python3.9/dist-packages/pipenv/core.py", line 269, in ensure_pipfile
    project.create_pipfile(python=python)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/project.py", line 710, in create_pipfile
    from .vendor.pip_shims.shims import InstallCommand
ImportError: cannot import name 'InstallCommand' from 'pipenv.vendor.pip_shims.shims' (/usr/local/lib/python3.9/dist-packages/pipenv/vendor/pip_shims/shims.py)

On our CI environment, we run pipenv lock [--dev-only] --requirements which breaks in a similar manner, loading the shim for Link; the end of that traceback looks like this:

  File "/usr/local/lib/python3.9/dist-packages/pipenv/core.py", line 1255, in do_init
    do_install_dependencies(
  File "/usr/local/lib/python3.9/dist-packages/pipenv/core.py", line 790, in do_install_dependencies
    deps_list = list(lockfile.get_requirements(dev=dev, only=dev_only))
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/requirementslib/models/lockfile.py", line 275, in get_requirements
    yield Requirement.from_pipfile(k, v)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2745, in from_pipfile
    r = FileRequirement.from_pipfile(name, pipfile)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/requirementslib/models/requirements.py", line 1794, in from_pipfile
    link_info = cls.get_link_from_line(uri)
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/requirementslib/models/requirements.py", line 1551, in get_link_from_line
    link = create_link(
  File "/usr/local/lib/python3.9/dist-packages/pipenv/vendor/requirementslib/models/utils.py", line 120, in create_link
    return Link(link)
TypeError: 'NoneType' object is not callable

Up until version 2021.5.29, things were working without issue.

Steps to replicate

I can reproduce the issue the easiest in a docker container; the following commands pull Ubuntu 20.04 and install python 3.9 from deadsnakes. It'll ask to set up a timezone in the process:

docker run --rm -it ubuntu:20.04 /bin/bash
apt update
apt install -y software-properties-common
add-apt-repository -y ppa:deadsnakes/ppa
apt update
apt install -y python3.9 python3.9-distutils python3-pip
python3.9 -m pip install pipenv

This sets you up with:

# python3.9 -m pip -V
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.9)
# pipenv --version
/usr/share/python-wheels/requests-2.22.0-py2.py3-none-any.whl/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.6) or chardet (3.0.4) doesn't match a supported version!
pipenv, version 2021.11.5.post0

and running pipenv lock at that point raises the exception.

---

$ pipenv --support

Pipenv version: '2021.11.5.post0'

Pipenv location: '/usr/local/lib/python3.9/dist-packages/pipenv'

Python location: '/usr/bin/python3.9'

Python installations found:

• 3.9.7: /usr/bin/python3.9
• 3.9.7: /bin/python3.9
• 3.8.10: /usr/bin/python3.8
• 3.8.10: /usr/bin/python3
• 3.8.10: /bin/python3.8
• 3.8.10: /bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.9.7',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.10.47-linuxkit',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Sat Jul 3 21:51:47 UTC 2021',
 'python_full_version': '3.9.7',
 'python_version': '3.9',
 'sys_platform': 'linux'}

System environment variables:

• HOSTNAME
• PWD
• HOME
• LS_COLORS
• TERM
• SHLVL
• PATH
• _
• LC_CTYPE
• PIP_SHIMS_BASE_MODULE
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
• PWD: /

---

[mjpieters]

Also of note: the RequestsDependencyWarning: urllib3 (1.26.6) or chardet (3.0.4) doesn't match a supported version! output only appears with this version of pipenv; it is gone when downgrading to 2021.5.29.

[TimidRobot]

GitHub Actions Failures

GitHub Actions currently uses Ubuntu 20.04, by default. Version 2021.11.5.post0 breaks GitHub Actions in multiple ways (depending on whether --system is used or not)

Pipenv 2021.11.5.post0 Environment Issues (without --system)

With our .github/workflows/pre-commit.yml, GitHub Actions fails with the following error:

Traceback (most recent call last):
  File "./manage.py", line 8, in <module>
    from django.conf import settings
ModuleNotFoundError: No module named 'django'

This was an unexpected error and it took me some time to narrow it down to issues with pipenv-2021.11.5.post0.

With the following diff, it continues working as it has for quite a while:

--- .github/workflows/pre-commit.yml
+++ .github/workflows/pre-commit.yml
@@ -61,9 +61,12 @@ jobs:
         python-version: "3.7"
 
     - name: Install pipenv
+      # Use pipenv 2021.5.29 to circumvent errors with 2021.11.5.post0 on
+      # Ubuntu 20.04
+      # https://github.com/pypa/pipenv/issues/4833
       run: |
         python -m pip install --upgrade pip
-        python -m pip install pipenv
+        python -m pip install pipenv==2021.5.29
 
     - name: Install Python dependencies
       run: |

Pipenv 2021.11.5.post0 Sync Issues (with --system)

Before I narrowed the issue down to pipenv-2021.11.5.post0, I tried changing our GitHub Action workflow to use --system. This caused a different error that was also solved by using pipenv-2021.5.29 (the diff above also resolved the following error:

Run pipenv sync --dev --system
Installing dependencies from Pipfile.lock (554a94)...

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.7.12/x64/bin/pipenv", line 8, in <module>
    sys.exit(cli())
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/vendor/click/decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/vendor/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/vendor/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/cli/command.py", line 683, in sync
    system=state.system
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/core.py", line 2848, in do_sync
    system=system,
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/core.py", line 1264, in do_init
    pypi_mirror=pypi_mirror,
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/core.py", line 819, in do_install_dependencies
    project, normal_deps, procs, failed_deps_queue, requirements_dir, **install_kwargs
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/core.py", line 683, in batch_install
    dep for dep in deps_to_install if not project.environment.is_satisfied(dep)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/core.py", line 683, in <listcomp>
    dep for dep in deps_to_install if not project.environment.is_satisfied(dep)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/project.py", line 349, in environment
    self._environment = self.get_environment(allow_global=allow_global)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/project.py", line 335, in get_environment
    pipfile=self.parsed_pipfile, project=self
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/environment.py", line 72, in __init__
    self._base_paths = self.get_paths()
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/environment.py", line 360, in get_paths
    c = subprocess_run(command)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/site-packages/pipenv/utils.py", line 2301, in subprocess_run
    encoding=encoding, **other_kwargs
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/subprocess.py", line 488, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/subprocess.py", line 800, in __init__
    restore_signals, start_new_session)
  File "/opt/hostedtoolcache/Python/3.7.12/x64/lib/python3.7/subprocess.py", line 1551, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/.local/share/virtualenvs/cc-licenses-aPyrXwUn/bin/python': '/home/runner/.local/share/virtualenvs/cc-licenses-aPyrXwUn/bin/python'
Error: Process completed with exit code 1.

[ilyalabun]

Got the same error No module named ... as @TimidRobot , but on CentOS 7 (not sure if OS is relevant here though) and we're running pipenv install and not pipenv sync --dev. Surprisingly, when we run pipenv sync all dependencies are installed in the right place and are discoverable by interpreter

[frostming]

Fixed by 4d1df95

[frostming]

Released: v2021.11.9

[ilyalabun]

Got the same error No module named ... as @TimidRobot , but on CentOS 7 (not sure if OS is relevant here though) and we're running pipenv install and not pipenv sync --dev. Surprisingly, when we run pipenv sync all dependencies are installed in the right place and are discoverable by interpreter

Just for the record -- apparently this issue is the separate one and it is not fixed in v2021.11.9, so I had to pin pipenv version as @TimidRobot did

[sbrunner]

Thanks :-)