Pre-commit hook with black

[sander76]

On my win10 machine I am trying to have my code formatted using black in a pre-commit hook.
However, it fails with the following error:

> git commit --quiet --allow-empty-message --file -
black....................................................................Failed
hookid: black

Traceback (most recent call last):
  File "C:\Users\sander\AppData\Local\Programs\Python\Python37\Lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\sander\AppData\Local\Programs\Python\Python37\Lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\sander\.cache\pre-commit\repoh3ogl_kr\py_env-python3.7\Scripts\black.exe\__main__.py", line 5, in <module>
  File "c:\users\sander\.cache\pre-commit\repoh3ogl_kr\py_env-python3.7\lib\site-packages\black.py", line 1, in <module>
    import asyncio
  File "C:\Users\sander\AppData\Local\Programs\Python\Python37\Lib\asyncio\__init__.py", line 8, in <module>
    from .base_events import *
  File "C:\Users\sander\AppData\Local\Programs\Python\Python37\Lib\asyncio\base_events.py", line 39, in <module>
    from . import events
  File "C:\Users\sander\AppData\Local\Programs\Python\Python37\Lib\asyncio\events.py", line 14, in <module>
    import contextvars
  File "C:\Users\sander\AppData\Local\Programs\Python\Python37\Lib\contextvars.py", line 1, in <module>
    from _contextvars import Context, ContextVar, Token, copy_context
ModuleNotFoundError: No module named '_contextvars'

My .pre-commit-config.yaml looks like this:

repos:
-   repo: https://github.com/ambv/black
    rev: stable
    hooks:
    - id: black
      language_version: python3.7

Anyone an idea?

[Caligatio]

Are you running a custom version of Python 3.7? It looks like Visual Studio compiled versions of Python had this problem but it was fixed in Feb 2019.

[sander76]

No just a version I installed from python.org (3.7.3)
First I thought it had to do with the fact that I was using Pycharm. But the same happens when using vscode (or command line)

[asottile]

hmmm this looks like you had 3.7.1 and then upgraded to 3.7.3 maybe? can you see if this persists after a pre-commit clean (usually shouldn't be necessary due to a healthy() check on the venv, but this module probably slipped through).

[sander76]

Unfortunately pre-commit clean did not help. I gave it a go with python3.6 which works.
If you like I can dig a bit deeper on suggestions you might have, but otherwise feel free to close this one as a one-off incident...

[asottile]

that's very very strange! does path/to/python37/python.exe -c 'import asyncio' work for you? I wonder if your installation is corrupted somehow 🤔

[sander76]

import asyncio works. Not sure how to check whether my python installation is corrupted. I haven't had any problems with it before.
I tried something else:
I removed the virtualenv I had and recreated one. I now got the following error:
I am going to reinstall python

(venv) C:\temp\p37>git commit -m tttes
[INFO] Installing environment for https://github.com/ambv/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: Command: ('c:\\temp\\p37\\venv\\scripts\\python.exe', '-mvirtualenv', 'C:\\Users\\sander\\.cache\\pre-commit\\repo8_loor_m\\py_env-python3.7', '-p', 'c:\\temp\\p37\\venv\\scripts\\python.exe')
Return code: 100
Expected return code: 0
Output:
    Already using interpreter c:\temp\p37\venv\scripts\python.exe
    Using base prefix 'C:\\Users\\sander\\AppData\\Local\\Programs\\Python\\Python37'
    New python executable in C:\Users\sander\.cache\pre-commit\repo8_loor_m\py_env-python3.7\Scripts\python.exe
    ERROR: The executable C:\Users\sander\.cache\pre-commit\repo8_loor_m\py_env-python3.7\Scripts\python.exe is not functioning
    ERROR: It thinks sys.prefix is 'c:\\' (should be 'c:\\users\\sander\\.cache\\pre-commit\\repo8_loor_m\\py_env-python3.7')
    ERROR: virtualenv is not compatible with this system or executable
    Note: some Windows users have reported this error when they installed Python for "Only this user" or have multiple versions of Python installed. Copying the appropriate PythonXX.dll to the virtualenv Scripts/ directory may fix this problem.

Errors:
    No pyvenv.cfg file

[asottile]

Hmm! Maybe this is the hint:

Note: some Windows users have reported this error when they installed Python for "Only this user" or have multiple

Try uninstalling python3.7 and then reinstalling with "for all users"?

[sander76]

Hmm.. Yeah. Did a re-install of python. for "all users". Error still there. I do have multiple python versions installed. But I don't want to mess with that.
Any clues ? Happy to help if you want me to try stuff. Otherwise I'm giving up. I'll retry when a new python version arrives. Thanks!

[asottile]

that's very very strange 🤔 -- one idea though, how did you create your outer venv (and did you recreate that after reinstalling python?)

[sander76]

I now did a clean installation of a fake project.
See below the steps I've taken.

C:\temp\p7>python -m venv venv

C:\temp\p7>venv\Scripts\activate

(venv) C:\temp\p7>pip install pre-commit

Collecting pre-commit
  Using cached https://files.pythonhosted.org/packages/d5/81/83ba6ac96d03c12ddbe987a177651ee9e35cc2a1567cc317d2556ac4acf4/pre_commit-1.16.1-py2.py3-none-any.whl
Collecting aspy.yaml (from pre-commit)
  Using cached https://files.pythonhosted.org/packages/99/ce/78be097b00817ccf02deaf481eb7a603eecee6fa216e82fa7848cd265449/aspy.yaml-1.3.0-py2.py3-none-any.whl
Collecting cfgv>=1.4.0 (from pre-commit)
  Downloading https://files.pythonhosted.org/packages/2f/ec/3c0a56fbc00e6b649c1dc809dc3f12c5796fbfb7940d1167b9bddc67b818/cfgv-2.0.0-py2.py3-none-any.whl
Collecting importlib-metadata (from pre-commit)
  Using cached https://files.pythonhosted.org/packages/63/da/4270fe75ba542de6a350c59339988e4a5275d3efff80b704ca2b5ede8120/importlib_metadata-0.15-py2.py3-none-any.whl
Collecting nodeenv>=0.11.1 (from pre-commit)
  Using cached https://files.pythonhosted.org/packages/00/6e/ed417bd1ed417ab3feada52d0c89ab0ed87d150f91590badf84273e047c9/nodeenv-1.3.3.tar.gz
Collecting virtualenv>=15.2 (from pre-commit)
  Using cached https://files.pythonhosted.org/packages/ca/ee/8375c01412abe6ff462ec80970e6bb1c4308724d4366d7519627c98691ab/virtualenv-16.6.0-py2.py3-none-any.whl
Collecting pyyaml (from pre-commit)
  Using cached https://files.pythonhosted.org/packages/b6/74/c3e15707516f80192059dde32c9d78f00a88c9f3c4d1efd76cbc8d5b0a20/PyYAML-5.1-cp37-cp37m-win_amd64.whl
Collecting toml (from pre-commit)
  Using cached https://files.pythonhosted.org/packages/a2/12/ced7105d2de62fa7c8fb5fce92cc4ce66b57c95fb875e9318dba7f8c5db0/toml-0.10.0-py2.py3-none-any.whl
Collecting six (from pre-commit)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting identify>=1.0.0 (from pre-commit)
  Using cached https://files.pythonhosted.org/packages/4a/91/73aefd8ef3cdb9470393f688987ae1b5edccf8923c9182ca74100bcbd725/identify-1.4.3-py2.py3-none-any.whl
Collecting zipp>=0.5 (from importlib-metadata->pre-commit)
  Using cached https://files.pythonhosted.org/packages/a0/0f/9bf71d438d2e9d5fd0e4569ea4d1a2b6f5a524c234c6d221b494298bb4d1/zipp-0.5.1-py2.py3-none-any.whl
Installing collected packages: pyyaml, aspy.yaml, six, cfgv, zipp, importlib-metadata, nodeenv, virtualenv, toml, identify, pre-commit
  Running setup.py install for nodeenv ... done
Successfully installed aspy.yaml-1.3.0 cfgv-2.0.0 identify-1.4.3 importlib-metadata-0.15 nodeenv-1.3.3 pre-commit-1.16.1 pyyaml-5.1 six-1.12.0 toml-0.10.0 virtualenv-16.6.0 zipp-0.5.1
You are using pip version 19.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

(venv) C:\temp\p7>git init
Initialized empty Git repository in C:/temp/p7/.git/

(venv) C:\temp\p7>dir
 Volume in drive C is OS
 Volume Serial Number is E660-FD09

 Directory of C:\temp\p7

05/28/2019  08:40    <DIR>          .
05/28/2019  08:40    <DIR>          ..
05/28/2019  08:39                 5 .gitignore
05/28/2019  08:40               124 .pre-commit-config.yaml
05/28/2019  08:40    <DIR>          .pytest_cache
05/28/2019  08:40                21 start.py
05/28/2019  08:37    <DIR>          venv
               3 File(s)            150 bytes
               4 Dir(s)  62,829,105,152 bytes free



(venv) C:\temp\p7>pre-commit install
pre-commit installed at .git\hooks\pre-commit

(venv) C:\temp\p7>git add .
warning: LF will be replaced by CRLF in start.py.
The file will have its original line endings in your working directory.

(venv) C:\temp\p7>git commit -m first_try
[INFO] Installing environment for https://github.com/ambv/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: Command: ('c:\\temp\\p7\\venv\\scripts\\python.exe', '-mvirtualenv', 'C:\\Users\\sander\\.cache\\pre-commit\\repougor8hmp\\py_env-python3.7', '-p', 'c:\\temp\\p7\\venv\\scripts\\python.exe')
Return code: 100
Expected return code: 0
Output:
    Already using interpreter c:\temp\p7\venv\scripts\python.exe
    Using base prefix 'C:\\Program Files\\Python37'
    New python executable in C:\Users\sander\.cache\pre-commit\repougor8hmp\py_env-python3.7\Scripts\python.exe
    ERROR: The executable C:\Users\sander\.cache\pre-commit\repougor8hmp\py_env-python3.7\Scripts\python.exe is not functioning
    ERROR: It thinks sys.prefix is 'c:\\' (should be 'c:\\users\\sander\\.cache\\pre-commit\\repougor8hmp\\py_env-python3.7')
    ERROR: virtualenv is not compatible with this system or executable
    Note: some Windows users have reported this error when they installed Python for "Only this user" or have multiple versions of Python installed. Copying the appropriate PythonXX.dll to the virtualenv Scripts/ directory may fix this problem.

Errors:
    No pyvenv.cfg file


Check the log at C:\Users\sander/.cache\pre-commit\pre-commit.log

(venv) C:\temp\p7>

[asottile]

hmmm maybe @gaborbernat has some hints here -- the virtualenv-in-venv support in virtualenv is pretty new

Can you try and see if this is a virtualenv issue? Here's the commands I tried:

$ python --version
Python 3.7.3

$ python -mvenv venv

$ venv\Scripts\pip.exe install virtualenv
Collecting virtualenv
  Downloading https://files.pythonhosted.org/packages/ca/ee/8375c01412abe6ff462ec80970e6bb1c4308724d4366d7519627c98691ab/virtualenv-16.6.0-py2.py3-none-any.whl (2.0MB)
Installing collected packages: virtualenv
Successfully installed virtualenv-16.6.0
You are using pip version 19.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

$ venv\Scripts\virtualenv.exe venv2
Using base prefix 'C:\\Python37'
New python executable in C:\Users\IEUser\AppData\Local\Temp\t\venv2\Scripts\python.exe
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter C:\Python37\python.exe

oh, I wonder if this is because your install of python has spaces in the path? Where is your python3.6 installed to?

[sander76]

I guess this all goes well.

C:\temp>python --version
Python 3.7.3

C:\temp>python -m venv venv

C:\temp>cd venv/Scripts

C:\temp\venv\Scripts>pip.exe install virtualenv
Collecting virtualenv
  Using cached https://files.pythonhosted.org/packages/ca/ee/8375c01412abe6ff462ec80970e6bb1c4308724d4366d7519627c98691ab/virtualenv-16.6.0-py2.py3-none-any.whl
Installing collected packages: virtualenv
Successfully installed virtualenv-16.6.0
You are using pip version 19.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

C:\temp\venv\Scripts>virtualenv.exe venv2
Running virtualenv with interpreter C:\Program Files\Python37\python.exe
Using base prefix 'C:\\Program Files\\Python37'
New python executable in C:\temp\venv\Scripts\venv2\Scripts\python.exe
Installing setuptools, pip, wheel...
done.

C:\temp\venv\Scripts>

It was installed in program files. I'll re-install in `c:\Python37\

[sander76]

Darn. installed python in c:\Python37. Same error. No idea what could be wrong.
I give up. Thanks for the help !

[asottile]

:( bummer we couldn't figure it out, these platform differences can be quite frustrating

I'd be willing to set up a call (google hangouts, discord, etc.) if you want to debug further -- I imagine we'd get to the bottom of it faster than back-and-forthing on github

[asottile]

if you want to debug this further, send me an email -- my email is attached to my commits (I'd post it here, but then the scrapers would get me more so than they already do!)

[Caligatio]

I think I found the problem after finding myself in this situation: venv vs. virtualenv.

If you create the "outer" virtualenv with python -m venv, the pre-commit/black install fails with the above error message. If you use python -m virtualenv (which may require installing virtualenv), the pre-commit/black install works as expected.

I unfortunately don't know enough about either venv or virtualenv to begin troubleshooting the problem.

P.S. - It appears PyCharm uses "venv" when it makes its virtual environments.

[asottile]

I'm able to reproduce this on one of my computers now:

Anthony@AnthonysDesktop MINGW64 /tmp/t/x (master)
$ git commit --allow-empty -mhi
[INFO] Initializing environment for https://github.com/python/black.
[INFO] Installing environment for https://github.com/python/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: Command: ('c:\\users\\anthony\\appdata\\local\\temp\\t\\x\\vvv\\scripts\\python.exe', '-mvirtualenv', 'C:\\Users\\Anthony\\.cache\\pre-commit\\reponhpojbe7\\py_env-python3', '-p', 'c:\\users\\anthony\\appdata\\local\\temp\\t\\x\\vvv\\scripts\\python.exe')
Return code: 100
Expected return code: 0
Output:
    Already using interpreter c:\users\anthony\appdata\local\temp\t\x\vvv\scripts\python.exe
    Using base prefix 'C:\\python37'
    New python executable in C:\Users\Anthony\.cache\pre-commit\reponhpojbe7\py_env-python3\Scripts\python.exe
    ERROR: The executable C:\Users\Anthony\.cache\pre-commit\reponhpojbe7\py_env-python3\Scripts\python.exe is not functioning
    ERROR: It thinks sys.prefix is 'c:\\' (should be 'c:\\users\\anthony\\.cache\\pre-commit\\reponhpojbe7\\py_env-python3')
    ERROR: virtualenv is not compatible with this system or executable
    Note: some Windows users have reported this error when they installed Python for "Only this user" or have multiple versions of Python installed. Copying the appropriate PythonXX.dll to the virtualenv Scripts/ directory may fix this problem.

Errors:
    No pyvenv.cfg file


Check the log at C:\Users\Anthony/.cache\pre-commit\pre-commit.log

[Caligatio]

It appears that the "-p /path/to/venv/created/virtualenv/python.exe" bit is causing the issue. I just left that off completely and the command worked.

[asottile]

ah yes, it's not discovering the original python correctly 🤔 -- we have code for this but it must've broken with the venv changes in 3.7.2 / 3.7.3(???)

[asottile]

looks like this is an interaction between virtualenv and the changes made in python 3.7.2 -- fortunately it looks like upstream is working on a fix for this! here's the fix for that

thanks again for the issue and being patient with my back and forth

[sander76]

@asottile Thanks for digging futher ! But not sure what this means. Should I wait with using this till python 3.7.4 is released ?

[Caligatio]

The problem actually is with the virtualenv package and they fixed the bug ~13 days ago. It looks like @asottile's comment on the bug ticket motivated them to hopefully release virtualenv 16.6.1 some time today.

[sander76]

But that's with virtualenv. Not venv?

[Caligatio]

So we have the outer "virtualenv" which can be made with venv or virtualenv and the inner "virtualenv" that is always made by pre-commit with virtualenv. The problem we hit was that virtualenv (used by pre-commit) does not nest well under venv. virtualenv is releasing a patch that will allow it to nest under venv.

[gaborbernat]

argh Azure DevOps still broken, breaks the automatic release of virtualenv 😓

[asottile]

oh lol sorry my comment above is confusing, I meant to include a link but was most likely distracted by stream chat 😆

This is the link that fixes things: pypa/virtualenv#1364

@gaborbernat is the virtualenv maintainer btw and it looks like 16.6.1 is ~attempting to be released but blocked by the annoying _sqlite3 azure pipelines bug

[gaborbernat]

bypassed by doing a manual release https://pypi.org/project/virtualenv/16.6.1/

[Caligatio]

Just tried it and virtualenv v16.6.1 fixes the bug. Thanks for the quick turnaround @asottile and @gaborbernat !

[asottile]

And thank you for the persistence and the report! This is very helpful for python users on windows!

[erobbed]

Hey @asottile and @gaborbernat - so I'm having what seems to be a similar issue to the above when trying to run pre-commit in my virtual environment. I'm running on virtualenv 16.6.1 already but still get the following error:

  File "/Users/ethan.roberts/Library/Caches/pypoetry/virtualenvs/my-project-py3.7/bin/pre-commit", line 6, in <module>
    from pre_commit.main import main
  File "/Users/ethan.roberts/Library/Caches/pypoetry/virtualenvs/my-project-py3.7/lib/python3.7/site-packages/pre_commit/main.py", line 12, in <module>
    from pre_commit.commands.autoupdate import autoupdate
  File "/Users/ethan.roberts/Library/Caches/pypoetry/virtualenvs/my-project-py3.7/lib/python3.7/site-packages/pre_commit/commands/autoupdate.py", line 15, in <module>
    from pre_commit.clientlib import CONFIG_SCHEMA
  File "/Users/ethan.roberts/Library/Caches/pypoetry/virtualenvs/my-project-py3.7/lib/python3.7/site-packages/pre_commit/clientlib.py", line 15, in <module>
    from pre_commit.error_handler import FatalError
  File "/Users/ethan.roberts/Library/Caches/pypoetry/virtualenvs/my-project-py3.7/lib/python3.7/site-packages/pre_commit/error_handler.py", line 13, in <module>
    from pre_commit.store import Store
  File "/Users/ethan.roberts/Library/Caches/pypoetry/virtualenvs/my-project-py3.7/lib/python3.7/site-packages/pre_commit/store.py", line 7, in <module>
    import sqlite3
  File "/Users/ethan.roberts/.pyenv/versions/3.7.3/lib/python3.7/sqlite3/__init__.py", line 23, in <module>
    from sqlite3.dbapi2 import *
  File "/Users/ethan.roberts/.pyenv/versions/3.7.3/lib/python3.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'

[asottile]

This means that the build of python from source was incorrectly done - - missing the sqlite extension. Since you're on macos I think that means you need to rebuild after getting sqlite from brew

[erobbed]

Thank you @asottile !! It seems I had a corrupted build in my pyenv and just had to uninstall that version and reinstall it to rebuild it like you said.