New version of virtualenv breaks everything

[iconmaster5326]

A few hours ago, a new version of virtualenv got pushed up: https://pypi.org/project/virtualenv/20.0.1/#history. This version breaks pre-commit.

When you install pre-commit in an empty Python environment via pip install pre-commit, the following happens. Any python hook will fail during the virtual environment setup, with the following error:

ImportError: cannot import name 'ensure_text'

This is related to pypa/virtualenv#1551. Said issue was marked as "not a bug" and closed, so it looks like its up to the dependees of virtualenv to update themselves to handle whatever changes caused this.

I'll attach my pre-commit.log shortly.

[asottile]

@gaborbernat

[iconmaster5326]

Here's what pip installs when I install pre-commit:

# pip3 install pre-commit
Collecting pre-commit
  Downloading https://files.pythonhosted.org/packages/ee/e8/5972cb8f5c52970ad2f841c0efd7cb6f031babfd85a77d3f90af87c613f3/pre_commit-2.0.1-py2.py3-none-any.whl (170kB)
    100% |################################| 174kB 3.9MB/s 
Collecting virtualenv>=15.2 (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/78/d3/d3d63f7ef35ff1fd0c47245721fd285a6a0f7ae03dda17a84d50a610affa/virtualenv-20.0.1-py2.py3-none-any.whl (4.6MB)
    100% |################################| 4.6MB 316kB/s 
Collecting importlib-metadata; python_version < "3.8" (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/8b/03/a00d504808808912751e64ccf414be53c29cad620e3de2421135fcae3025/importlib_metadata-1.5.0-py2.py3-none-any.whl
Collecting importlib-resources; python_version < "3.7" (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/2f/f7/b4aa02cdd3ee7ebba375969d77c00826aa15c5db84247d23c89522dccbfa/importlib_resources-1.0.2-py2.py3-none-any.whl
Collecting toml (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/a2/12/ced7105d2de62fa7c8fb5fce92cc4ce66b57c95fb875e9318dba7f8c5db0/toml-0.10.0-py2.py3-none-any.whl
Collecting identify>=1.0.0 (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/73/08/ebd9cd04a6741b2cdf9a634170070d16085a38c2041e11a9b634c1cef623/identify-1.4.11-py2.py3-none-any.whl (97kB)
    100% |################################| 102kB 9.1MB/s 
Collecting aspy.yaml (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/99/ce/78be097b00817ccf02deaf481eb7a603eecee6fa216e82fa7848cd265449/aspy.yaml-1.3.0-py2.py3-none-any.whl
Collecting cfgv>=2.0.0 (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/6e/ff/2e6bcaff26058200717c469a0910da96c89bb00e9cc31b68aa0bfc9b1b0d/cfgv-2.0.1-py2.py3-none-any.whl
Collecting nodeenv>=0.11.1 (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/08/43/86ff33286c83f7b5e8903c32db01fe122c5e8a9d8dc1067dcaa9be54a033/nodeenv-1.3.5-py2.py3-none-any.whl
Collecting pyyaml (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/3d/d9/ea9816aea31beeadccd03f1f8b625ecf8f645bd66744484d162d84803ce5/PyYAML-5.3.tar.gz (268kB)
    100% |################################| 276kB 4.3MB/s 
Collecting appdirs<2,>=1.4.3 (from virtualenv>=15.2->pre-commit)
  Downloading https://files.pythonhosted.org/packages/56/eb/810e700ed1349edde4cbdc1b2a21e28cdf115f9faf263f6bbf8447c1abf3/appdirs-1.4.3-py2.py3-none-any.whl
Collecting six<2,>=1.12.0 (from virtualenv>=15.2->pre-commit)
  Downloading https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl
Collecting filelock<4,>=3.0.0 (from virtualenv>=15.2->pre-commit)
  Downloading https://files.pythonhosted.org/packages/93/83/71a2ee6158bb9f39a90c0dea1637f81d5eef866e188e1971a1b1ab01a35a/filelock-3.0.12-py3-none-any.whl
Collecting zipp>=0.5 (from importlib-metadata; python_version < "3.8"->pre-commit)
  Downloading https://files.pythonhosted.org/packages/46/42/f2dd964b2a6b1921b08d661138148c1bcd3f038462a44019416f2342b618/zipp-2.2.0-py36-none-any.whl
Building wheels for collected packages: pyyaml
  Running setup.py bdist_wheel for pyyaml ... done
  Stored in directory: /root/.cache/pip/wheels/e4/76/4d/a95b8dd7b452b69e8ed4f68b69e1b55e12c9c9624dd962b191
Successfully built pyyaml
Installing collected packages: zipp, importlib-metadata, appdirs, six, filelock, importlib-resources, virtualenv, toml, identify, pyyaml, aspy.yaml, cfgv, nodeenv, pre-commit
  Found existing installation: six 1.11.0
    Not uninstalling six at /usr/lib/python3/dist-packages, outside environment /usr
Successfully installed appdirs-1.4.3 aspy.yaml-1.3.0 cfgv-2.0.1 filelock-3.0.12 identify-1.4.11 importlib-metadata-1.5.0 importlib-resources-1.0.2 nodeenv-1.3.5 pre-commit-2.0.1 pyyaml-5.3 six-1.14.0 toml-0.10.0 virtualenv-20.0.1 zipp-2.2.0

[gaborbernat]

Seems six is pulled in from the global site-packages of Ubuntu folder and hence pip ignores the six requirement constraint... One can either bump six from Ubuntu or install pre-commit into an isolated environment as I see. Generally, it's not supported to mix Ubuntu python packages with packages from PyPi in the same python environment. Use virtual environments to create separation, or install pre-commit from Ubuntu package manager also. In the future, pypa/pip#988 would throw an exception, but for now, it silently ignores it. If this is your developer machine consider using https://pipxproject.github.io/pipx/installation/ to automatically create these isolated environments for tools.

[iconmaster5326]

Ah, good to know it's an issue with Ubuntu. Thank you for looking into it.

[asottile]

I've worked around this by making cfgv 3.x no longer depend on six, this now succeeds:

$ docker build -t test .
Sending build context to Docker daemon  2.048kB
Step 1/4 : FROM ubuntu:bionic
 ---> 549b9b86cb8d
Step 2/4 : RUN :     && apt-get update     && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends         python3-distutils         python3-pip         python3-setuptools         python3-six     && apt-get clean     && rm -rf /var/lib/apt/lists/*
 ---> Using cache
 ---> c944a9b38718
Step 3/4 : RUN pip3 install pre-commit
 ---> Running in 2c5c8ade9f32
Collecting pre-commit
  Downloading https://files.pythonhosted.org/packages/ee/e8/5972cb8f5c52970ad2f841c0efd7cb6f031babfd85a77d3f90af87c613f3/pre_commit-2.0.1-py2.py3-none-any.whl (170kB)
Collecting nodeenv>=0.11.1 (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/08/43/86ff33286c83f7b5e8903c32db01fe122c5e8a9d8dc1067dcaa9be54a033/nodeenv-1.3.5-py2.py3-none-any.whl
Collecting importlib-metadata; python_version < "3.8" (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/8b/03/a00d504808808912751e64ccf414be53c29cad620e3de2421135fcae3025/importlib_metadata-1.5.0-py2.py3-none-any.whl
Collecting toml (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/a2/12/ced7105d2de62fa7c8fb5fce92cc4ce66b57c95fb875e9318dba7f8c5db0/toml-0.10.0-py2.py3-none-any.whl
Collecting virtualenv>=15.2 (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/78/d3/d3d63f7ef35ff1fd0c47245721fd285a6a0f7ae03dda17a84d50a610affa/virtualenv-20.0.1-py2.py3-none-any.whl (4.6MB)
Collecting importlib-resources; python_version < "3.7" (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/2f/f7/b4aa02cdd3ee7ebba375969d77c00826aa15c5db84247d23c89522dccbfa/importlib_resources-1.0.2-py2.py3-none-any.whl
Collecting cfgv>=2.0.0 (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/2a/b8/1b9626e940bf80cc5a19a19e6b4c99282aa88f438cbcd2d86d7a2a666974/cfgv-3.0.0-py2.py3-none-any.whl
Collecting identify>=1.0.0 (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/73/08/ebd9cd04a6741b2cdf9a634170070d16085a38c2041e11a9b634c1cef623/identify-1.4.11-py2.py3-none-any.whl (97kB)
Collecting pyyaml (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/3d/d9/ea9816aea31beeadccd03f1f8b625ecf8f645bd66744484d162d84803ce5/PyYAML-5.3.tar.gz (268kB)
Collecting aspy.yaml (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/99/ce/78be097b00817ccf02deaf481eb7a603eecee6fa216e82fa7848cd265449/aspy.yaml-1.3.0-py2.py3-none-any.whl
Collecting zipp>=0.5 (from importlib-metadata; python_version < "3.8"->pre-commit)
  Downloading https://files.pythonhosted.org/packages/46/42/f2dd964b2a6b1921b08d661138148c1bcd3f038462a44019416f2342b618/zipp-2.2.0-py36-none-any.whl
Collecting filelock<4,>=3.0.0 (from virtualenv>=15.2->pre-commit)
  Downloading https://files.pythonhosted.org/packages/93/83/71a2ee6158bb9f39a90c0dea1637f81d5eef866e188e1971a1b1ab01a35a/filelock-3.0.12-py3-none-any.whl
Collecting six<2,>=1.12.0 (from virtualenv>=15.2->pre-commit)
  Downloading https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl
Collecting appdirs<2,>=1.4.3 (from virtualenv>=15.2->pre-commit)
  Downloading https://files.pythonhosted.org/packages/56/eb/810e700ed1349edde4cbdc1b2a21e28cdf115f9faf263f6bbf8447c1abf3/appdirs-1.4.3-py2.py3-none-any.whl
Building wheels for collected packages: pyyaml
  Running setup.py bdist_wheel for pyyaml: started
  Running setup.py bdist_wheel for pyyaml: finished with status 'error'
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-p8by_vr_/pyyaml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpcz4z3udspip-wheel- --python-tag cp36:
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help
  
  error: invalid command 'bdist_wheel'
  
  ----------------------------------------
  Failed building wheel for pyyaml
  Running setup.py clean for pyyaml
Failed to build pyyaml
Installing collected packages: nodeenv, zipp, importlib-metadata, toml, filelock, importlib-resources, six, appdirs, virtualenv, cfgv, identify, pyyaml, aspy.yaml, pre-commit
  Found existing installation: six 1.11.0
    Not uninstalling six at /usr/lib/python3/dist-packages, outside environment /usr
  Running setup.py install for pyyaml: started
    Running setup.py install for pyyaml: finished with status 'done'
Successfully installed appdirs-1.4.3 aspy.yaml-1.3.0 cfgv-3.0.0 filelock-3.0.12 identify-1.4.11 importlib-metadata-1.5.0 importlib-resources-1.0.2 nodeenv-1.3.5 pre-commit-2.0.1 pyyaml-5.3 six-1.14.0 toml-0.10.0 virtualenv-20.0.1 zipp-2.2.0
Removing intermediate container 2c5c8ade9f32
 ---> 0c289791e793
Step 4/4 : RUN virtualenv /venv
 ---> Running in ff482e4bed8b
Removing intermediate container ff482e4bed8b
 ---> 627844f28a42
Successfully built 627844f28a42
Successfully tagged test:latest

it'll still be broken for ubuntu xenial in all likelihood (system python3.5) but I can't do anything about that sadly

[ltalirz]

Is there a suggestion for what to do on CI platforms like travis that use ubuntu-based images?

Supposedly, travis already creates virtualenvs (and I have never had to create one manually before) but I still seem to hit this issue:
https://travis-ci.org/ltalirz/aiida-zeopp/jobs/649566989#L1134

[asottile]

I believe you can use six>=... manually? unclear though as pre-commit no longer depends on six except through virtualenv

[ltalirz]

Thanks for the super-quick reply. Explicitly requesting six>=1.12.0 worked for me.