Multiprocessing issue when redirecting stdout

[asottile]

In GitLab by @conkiztador on Dec 21, 2014, 19:59

Running 2.2.5 on linux, when I run flake8 on multiple files with a lot of violations and redirect the output at all (less, grep, >file) I often see junk in the output due to some issue with multiprocessing.

$ for i in `seq 0 1000`; do echo test$i >> testcase.py; done

$ .tox/flake8/bin/flake8 testcase.py testcase.py testcase.py | head -5
testcase.py:1:1: F821 undefined name 'test0'
testcase.py:2:1: F821 undefined name 'test1'
testcase.py:3:1: F821 undefined name 'test2'
testcase.py:4:1: F821 undefined name 'test3'

Grepping for junk:

$ .tox/flake8/bin/flake8 testcase.py testcase.py testcase.py | grep -E '^[^t]'
821 undefined name 'test422'
:256:1: F821 undefined name 'test255'
py:674:1: F821 undefined name 'test673'
 'test756'

$ .tox/flake8/bin/flake8 testcase.py testcase.py testcase.py | grep -E '^[^t]'
efined name 'test87'

$ .tox/flake8/bin/flake8 testcase.py testcase.py testcase.py | grep -E '^[^t]'
821 undefined name 'test422'
'

-j1 fixes it:

$ .tox/flake8/bin/flake8 -j1 testcase.py testcase.py testcase.py | grep -E '^[^t]'
$ .tox/flake8/bin/flake8 -j1 testcase.py testcase.py testcase.py | grep -E '^[^t]'
$ .tox/flake8/bin/flake8 -j1 testcase.py testcase.py testcase.py | grep -E '^[^t]'



$ .tox/flake8/bin/pip freeze
argparse==1.2.1
flake8==2.2.5
mccabe==0.3
pep8==1.5.7
pyflakes==0.8.1
six==1.8.0
wsgiref==0.1.2

[asottile]

In GitLab by @sigmavirus24 on Dec 22, 2014, 12:56

So I think this is actually entirely due to the output being interleaved. Thanks to your excellent steps to reproduce this, I can confirm that I see lines like

testcase.py:590:1: F821 undef821 undefined name 'test422'

And then later

ined name 'test589'

I'll have a look at how our reporters work and if there's a way to prevent this without hurting performance

[asottile]

In GitLab by @sigmavirus24 on Dec 26, 2014, 19:38

mentioned in merge request !15

[asottile]

In GitLab by @methane on Dec 27, 2014, 01:01

This can be happen because stdout is block buffered when not stdout.isatty().
os.write() is the most safe way.
But flush() for each line can be enough and easy to achieve Python 2.5~ compatibility.

[asottile]

In GitLab by @methane on Dec 27, 2014, 01:06

mentioned in merge request !16

[asottile]

In GitLab by @methane on Dec 27, 2014, 02:31

PyCQA/pycodestyle#363

[asottile]

In GitLab by @sigmavirus24 on Dec 27, 2014, 07:00

Status changed to closed by commit a239fe7

[asottile]

In GitLab by @sigmavirus24 on Dec 27, 2014, 07:04

Thanks for reporting this @conkiztador and your help @methane

[asottile]

In GitLab by @sigmavirus24 on Dec 27, 2014, 07:33

mentioned in commit 6b27451