pre-commit flake8 check fails with no error message

[drolando]

The pre-commit flake8 check is failing without any error message for me. It just says Failed.

pre-commit runtests: commands[1] | pre-commit run --all-files
Trim Trailing Whitespace.................................................Passed
Fix End of Files.........................................................Passed
Check docstring is first.................................................Passed
Check that executables have shebangs.................(no files to check)Skipped
Check for merge conflicts................................................Passed
Check Yaml...............................................................Passed
Debug Statements (Python)................................................Passed
Fix double quoted strings................................................Passed
Flake8...................................................................Failed
Check for added large files..............................................Passed
Check for byte-order marker..............................................Passed
Fix python encoding pragma...............................................Passed
Fix requirements.txt.....................................................Passed
Reorder python imports...................................................Passed
autopep8.................................................................Passed
pyupgrade................................................................Passed
ERROR: InvocationError: '/nail/home/drolando/pg/python/yelp_zipkin_utils/.tox/pre-commit/bin/pre-commit
 run --all-files'
_______________________________________________ summary _______________________________________________
  py27: commands succeeded
  py36: commands succeeded
  py37: commands succeeded
ERROR:   pre-commit: commands failed
Makefile:4: recipe for target 'test' failed
make: *** [test] Error 1

If I source the virtualenv and run flake8 everything works though, so there seems to be something special with the way pre-commit runs it.

drolando@dev128-uswest1adevc yelp_zipkin_utils (git master) ✓
~>  source .tox/pre-commit/bin/activate
drolando@dev128-uswest1adevc yelp_zipkin_utils (git master) ✓
~>  flake8
drolando@dev128-uswest1adevc yelp_zipkin_utils (git master) ✓
~>  

~>  cat .pre-commit-config.yaml
repos:
-   repo: git@git.yelpcorp.com:mirrors/pre-commit/pre-commit-hooks
    rev: v2.4.0
    hooks:
    -   id: trailing-whitespace
    -   id: end-of-file-fixer
    -   id: check-docstring-first
    -   id: check-executables-have-shebangs
    -   id: check-merge-conflict
    -   id: check-yaml
    -   id: debug-statements
    -   id: double-quote-string-fixer
    -   id: flake8
    -   id: check-added-large-files
    -   id: check-byte-order-marker
    -   id: fix-encoding-pragma
    -   id: requirements-txt-fixer

~>  cat tox.ini | grep -A 3 flake8
[flake8]
ignore =
exclude = .git,.tox,docs,virtualenv_run
filename = *.py
(pre-commit)

The interesting thing is that it works fine if I use python3.6, it only seems to fail if I use python3.7 (3.7.5). @asottile have you seen this issue before? Any suggestion on how to debug it otherwise? Adding -v doesn't print anything useful...

[asottile]

this means that flake8 is returning no output and exiting nonzero -- someone else hit this before in the tracker though: pre-commit/pre-commit#1221

it ended up being that python had upgraded out from under them and the interpreter was only partially functioning (but still passed the healthy() check?)

perhaps you can run the commands that I requested there and actually produce the output / get to a more helpful answer ;)

[asottile]

also note that you're using the deprecated flake8 hook

[drolando]

If I run for f in $(find . -name '*.py' | grep -vE '.tox|.git'); do echo $f; pre-commit run flake8 --file $f; done they all pass. It only breaks when using --all-files.
flake8; echo $? returns 0 correctly, so not sure what's up with it.

However rm -rf ~/.cache/pre-commit worked and now the flake8 hook is working fine. 🤷‍♂

[asottile]

you should use pre-commit clean instead for that rm -rf command -- but now our reproduction is gone :( -- I was hoping you'd try some of the commands against the flake8 cache executable in that issue before deleting :(

ah well, nothing we can do here now

[drolando]

@asottile I actually run mv ~/.cache/pre-commit ~/.cache/pre-commit.bak so we should be able to get back the old state

[drolando]

oh but there are 150 different folders in there named with a sha, so now I don't know anymore which one was it :(

[asottile]

there's a little sqlite database you can poke at db.db -- the format is intentionally undocumented implementation detail but you may be able to poke around at it a bit (it's basically keyed off of the repository name and language_version)

[drolando]

Swapping the folders again did the trick, now I have again the broken pre-commit env.
What did you want me to try?

~>  pre-commit run flake8 --all-files
Flake8...................................................................Failed
~>  which flake8
/nail/home/drolando/pg/python/yelp_zipkin_utils/.tox/pre-commit/bin/flake8
~>  flake8 --version
3.7.8 (mccabe: 0.6.1, pycodestyle: 2.5.0, pyflakes: 2.1.1) CPython 3.7.5 on Linux

[asottile]

I'm curious what invoking the flake8 executable inside the particular cached folder results in, you can pre-commit gc to clean up any unused ones and then you should be able to find it via $ ls ~/.cache/pre-commit/repo*/py_env*/bin/flake8

you can cross reference it against the database in ~/.cache/pre-commit/db.db

the (probably) important things to look at are the difference between python3.7 --version and path/to/env/bin/python --version (from the hook environments)

as well as trying the env flake8 with -vvv

[drolando]

Yup the python versions don't match

~>  /nail/home/drolando/.cache/pre-commit/repoi_yCPK/py_env-python3.7/bin/python --version
Python 3.7.3
~>  python3.7 --version
Python 3.7.5

And running flake8 from that virtualenv segfaults

drolando@dev128-uswest1adevc yelp_zipkin_utils (git master) ✓
~>  /nail/home/drolando/.cache/pre-commit/repoi_yCPK/py_env-python3.7/bin/python -m flake8
[1]    8317 segmentation fault  /nail/home/drolando/.cache/pre-commit/repoi_yCPK/py_env-python3.7/bin/python

Looks like python 3.7.5 was backported to bionic in November http://changelogs.ubuntu.com/changelogs/pool/universe/p/python3.7/python3.7_3.7.5-2~18.04/changelog. I thought ubuntu would never update versions outside of a new releast though...

[asottile]

yeah the ubuntu maintainer has been doing some very strange things with this LTS release 🤔

they've continued to update the 3.7.x packages and even introduced a 3.8 package, it's been quite a lot more work to keep up with these surprises in deadsnakes :(

but yeah I guess that explains it, given that segmentation fault. I'm kind of surprised the healthy() check passes, but perhaps it doesn't import the proper modules to cause a segfault

could you strace it and attach a log? that might be helpful for improving the healthy() check in pre-commit such that these are automagically rebuilt

[drolando]

strace: https://i.fluffy.cc/fp07X05FnxqQ7QtGsgpcxRPnDFflgZkW.html

The python executable is not entirely broken, I am able to get a shell...

[asottile]

huh yeah seems it only ends up segfaulting after trying to do multiprocessing things -- probably not something that can be cheaply detected in healthy() :/