Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Different output with --jobs=1 and --jobs=2 #374

Closed
pylint-bot opened this issue Nov 12, 2014 · 55 comments · Fixed by #4007, #8757 or #9093
Closed

Different output with --jobs=1 and --jobs=2 #374

pylint-bot opened this issue Nov 12, 2014 · 55 comments · Fixed by #4007, #8757 or #9093
Labels
Bug 🪲 High priority Issue with more than 10 reactions multiprocessing Needs PR This issue is accepted, sufficiently specified and now needs an implementation
Milestone

Comments

@pylint-bot
Copy link

Originally reported by: Robert Spier (BitBucket: robert_spier)


Parallelism (--jobs) changes the output of pylint.

It's not just the order of the tests, --jobs=2 outputs 18468 lines of output compared to only 21 for --jobs=1. pylint 1.3.1 reports no lint errors.

$ venv/bin/pylint --jobs=2 --rcfile=$PWD/pylintrc app/codein app/melange app/soc app/summerofcode app/settings.py app/urls.py app/main.py tests pavement.py setup.py 2>&1 | head
************* Module codein.callback
W: 17, 0: import missing from __future__ import absolute_import (no-absolute-import)
W: 18, 0: import missing from __future__ import absolute_import (no-absolute-import)
W: 19, 0: import missing from __future__ import absolute_import (no-absolute-import)
W: 20, 0: import missing from __future__ import absolute_import (no-absolute-import)
************* Module codein.types
W: 17, 0: import missing from __future__ import absolute_import (no-absolute-import)
W: 18, 0: import missing from __future__ import absolute_import (no-absolute-import)
W: 20, 0: import missing from __future__ import absolute_import (no-absolute-import)
W: 21, 0: import missing from __future__ import absolute_import (no-absolute-import)

$ venv/bin/pylint --jobs=1 --rcfile=$PWD/pylintrc app/codein app/melange app/soc app/summerofcode app/settings.py app/urls.py app/main.py tests pavement.py setup.py 2>&1 | head
************* Module main
E: 46, 2: print statement used (print-statement)
E: 47, 2: print statement used (print-statement)
E: 48, 2: print statement used (print-statement)
E: 49, 2: print statement used (print-statement)
E: 50, 2: print statement used (print-statement)
************* Module tests.test_utils
E:658, 8: print statement used (print-statement)
E:662,10: print statement used (print-statement)
E:667, 8: print statement used (print-statement)

This is with the current head of the pylint repository. (1889:e404dd4d6e37 tip)

The source code being linted is https://code.google.com/p/soc/


@pylint-bot
Copy link
Author

Original comment by Robert Spier (BitBucket: robert_spier):


And here's the output with formatting fixed.

$ venv/bin/pylint --jobs=2 --rcfile=$PWD/pylintrc app/codein app/melange app/soc app/summerofcode app/settings.py app/urls.py app/main.py tests pavement.py setup.py 2>&1 | head

#!text

************* Module codein.callback
W: 17, 0: import missing `from __future__ import absolute_import` (no-absolute-import)
W: 18, 0: import missing `from __future__ import absolute_import` (no-absolute-import)
W: 19, 0: import missing `from __future__ import absolute_import` (no-absolute-import)
W: 20, 0: import missing `from __future__ import absolute_import` (no-absolute-import)
************* Module codein.types
W: 17, 0: import missing `from __future__ import absolute_import` (no-absolute-import)
W: 18, 0: import missing `from __future__ import absolute_import` (no-absolute-import)
W: 20, 0: import missing `from __future__ import absolute_import` (no-absolute-import)
W: 21, 0: import missing `from __future__ import absolute_import` (no-absolute-import)

$ venv/bin/pylint --jobs=1 --rcfile=$PWD/pylintrc app/codein app/melange app/soc app/summerofcode app/settings.py app/urls.py app/main.py tests pavement.py setup.py 2>&1 | head

#!text

************* Module main
E: 46, 2: print statement used (print-statement)
E: 47, 2: print statement used (print-statement)
E: 48, 2: print statement used (print-statement)
E: 49, 2: print statement used (print-statement)
E: 50, 2: print statement used (print-statement)
************* Module tests.test_utils
E:658, 8: print statement used (print-statement)
E:662,10: print statement used (print-statement)
E:667, 8: print statement used (print-statement)
************* Module tests.run
E:471, 4: print statement used (print-statement)
E:473, 4: print statement used (print-statement)

@pylint-bot
Copy link
Author

Original comment by Robert Spier (BitBucket: robert_spier):


FYI, I can also replicate this with the official 1.4.0 release. Although the output is slightly different. Running with --jobs=2 produces many more lint warnings than with --jobs=1.

@pylint-bot
Copy link
Author

Original comment by Saulius Menkevičius (BitBucket: sauliusmenkevicius):


Can confirm.

For me, pylint seems to ignore the pylintrc file, even though it is specified via the --rcfile= option, when -j 2+ is set.

I was using an older version of pylint (with support for --jobs) from hg+http://bitbucket.org/godfryd/pylint/@763d12c3c923f0733fc5c1866c69d973475993cd#egg=pylint from this PR: https://bitbucket.org/logilab/pylint/pull-request/82/added-support-for-checking-files-in/commits; which seemed to respect --pylintrc properly in multi-process mode. Something broke inbetween that checkin and 1.4.0

@pylint-bot
Copy link
Author

Original comment by Pedro Algarvio (BitBucket: s0undt3ch, GitHub: @s0undt3ch?):


I can also confirm this. I had custom plugins on the rcfile and they're not being loaded

@pylint-bot
Copy link
Author

Original comment by Pedro Algarvio (BitBucket: s0undt3ch, GitHub: @s0undt3ch?):


I'm also on 1.4.0

@pylint-bot
Copy link
Author

Original comment by Claudiu Popa (BitBucket: PCManticore, GitHub: @PCManticore):


Yeah, --jobs is kind tricky right now, will try to fix asap. By the way, if you have any other problems with --jobs, it's better to open another ticket, so they could be tracked and fixed individually.

@pylint-bot
Copy link
Author

Original comment by Pedro Algarvio (BitBucket: s0undt3ch, GitHub: @s0undt3ch?):


I believe my problem is because the rcfile is being ignored.

@pylint-bot
Copy link
Author

Original comment by Michal Nowikowski (BitBucket: godfryd, GitHub: @godfryd?):


The issue has been fixed by pull request #213.

@pylint-bot
Copy link
Author

Original comment by Claudiu Popa (BitBucket: PCManticore, GitHub: @PCManticore):


Merged in godfryd/pylint/fix-374 (pull request #213)

Fixed passing configuration from master linter to sublinters. Closes issue #374.

1 similar comment
@pylint-bot
Copy link
Author

Original comment by Claudiu Popa (BitBucket: PCManticore, GitHub: @PCManticore):


Merged in godfryd/pylint/fix-374 (pull request #213)

Fixed passing configuration from master linter to sublinters. Closes issue #374.

@pylint-bot
Copy link
Author

Original comment by Claudiu Popa (BitBucket: PCManticore, GitHub: @PCManticore):


Fixed passing configuration from master linter to sublinters. Closes issue #374.

@pylint-bot
Copy link
Author

Original comment by Claudiu Popa (BitBucket: PCManticore, GitHub: @PCManticore):


Hm, the latest patch introduced another regression related to disabling messages from the pylintrc.

@pylint-bot
Copy link
Author

Original comment by Michal Nowikowski (BitBucket: godfryd, GitHub: @godfryd?):


How to reproduce that new problem?

I run:

  1. pylint --jobs=2 --rcfile=pylintrc app/codein app/melange app/soc app/summerofcode app/settings.py app/urls.py app/main.py tests pavement.py setup.py
  2. pylint --jobs=1 --rcfile=pylintrc app/codein app/melange app/soc app/summerofcode app/settings.py app/urls.py app/main.py tests pavement.py setup.py

The outputs contain the same messages. pylintrc is disabling and enabling particular messages.

@pylint-bot
Copy link
Author

Original comment by Claudiu Popa (BitBucket: PCManticore, GitHub: @PCManticore):


Indeed, it was due to a faulty installation. Sorry about the noise.

@pylint-bot
Copy link
Author

Original comment by Robert Spier (BitBucket: robert_spier):


I confirm that it works correctly on the Melange codebase. The performance improvement from adding more jobs is not as high as expected.

@pylint-bot
Copy link
Author

Original comment by Pavel Roskin (BitBucket: pavel_roskin):


The issue with absolute_import may be resolved by pull request #229. The performance is discussed in issue #479. There are still issues with different output. Not sure if I should open another ticket. I'll describe it here.

@pylint-bot
Copy link
Author

Original comment by Pavel Roskin (BitBucket: pavel_roskin):


That's a minimal example showing that the issue is not fully resolved.

#!shell

echo 'pass' > first.py
echo 'pass' > second.py
pylint first.py second.py >output1
pylint --jobs=2 first.py second.py >output2
diff -u output1 output2
#!diff

--- output1 2015-02-25 18:51:36.770036133 -0500
+++ output2 2015-02-25 18:51:39.274040857 -0500
@@ -6,7 +6,7 @@

 Report
 ======
-3 statements analysed.
+4 statements analysed.

 Statistics by type
 ------------------
@@ -72,31 +72,18 @@



-% errors / warnings by module
------------------------------
-
-+-------+------+--------+---------+-----------+
-|module |error |warning |refactor |convention |
-+=======+======+========+=========+===========+
-|second |0.00  |0.00    |0.00     |50.00      |
-+-------+------+--------+---------+-----------+
-|first  |0.00  |0.00    |0.00     |50.00      |
-+-------+------+--------+---------+-----------+
-
-
-
 Messages
 --------

 +------------------+------------+
 |message id        |occurrences |
 +==================+============+
-|missing-docstring |2           |
+|missing-docstring |1           |
 +------------------+------------+



 Global evaluation
 -----------------
-Your code has been rated at 3.33/10 (previous run: 3.33/10, +0.00)
+Your code has been rated at 5.00/10 (previous run: 3.33/10, +1.67)

@pylint-bot
Copy link
Author

Original comment by James Broadhead (BitBucket: jamesbroadhead, GitHub: @jamesbroadhead?):


Same here -- getting significantly more 'duplicate-code' & 'cyclic-import' with --jobs=1 over --jobs=2

Closed-source codebase, but I'm happy to run debug branches etc. over it.

$ pylint --version
pylint 1.4.4,
astroid 1.3.6, common 1.0.2
Python 2.7.8 (default, Sep 10 2014, 04:44:11)
[GCC 4.9.1]

@pylint-bot
Copy link
Author

Original comment by Claudiu Popa (BitBucket: PCManticore, GitHub: @PCManticore):


Increasing to blocker, so that we'll have a fix finally in 1.5.

@pylint-bot
Copy link
Author

Original comment by Claudiu Popa (BitBucket: PCManticore, GitHub: @PCManticore):


I'm trying to reproduce this issue, but unfortunately I can't reproduce Pavel's example using the latest code from the repository. Could anyone of you provide a more comprehensive example where this discrepancy happens?

@pylint-bot
Copy link
Author

Original comment by Pavel Roskin (BitBucket: pavel_roskin):


I tried pylint on the current python-ly, and I see that things have improved greatly. There are only two issues that the parallel pylint missed compared to the single-job version: R0401(cyclic-import) and R0801(duplicate-code). There is also a discrepancy in the way __init__.py is referred to.

One job:

************* Module ly
ly/__init__.py(46): [C0303(trailing-whitespace)] Trailing whitespace

Two jobs:

************* Module ly.__init__
ly/__init__.py(46): [C0303(trailing-whitespace)] Trailing whitespace

I actually prefer the later notation, it's more explicit that it's just the __init__.py file, not the whole module.

@pylint-bot
Copy link
Author

Original comment by Pavel Roskin (BitBucket: pavel_roskin):


Here's a simple test for the module naming issue. It turns out the parallel version uses both names, which is bad.

[proski@dell pylinttest]$ echo "pass " >__init__.py
[proski@dell pylinttest]$ pylint -r n --jobs=1 .
************* Module pylinttest
C:  1, 0: Trailing whitespace (trailing-whitespace)
C:  1, 0: Missing module docstring (missing-docstring)
[proski@dell pylinttest]$ pylint -r n --jobs=2 .
************* Module pylinttest.__init__
C:  1, 0: Trailing whitespace (trailing-whitespace)
************* Module pylinttest
C:  1, 0: Missing module docstring (missing-docstring)

@pylint-bot
Copy link
Author

Original comment by Pavel Roskin (BitBucket: pavel_roskin):


Cyclic import problem

:::text
[proski@dell pylintcycle]$ touch __init__.py
[proski@dell pylintcycle]$ echo 'import pylintcycle.second' >first.py
[proski@dell pylintcycle]$ echo 'import pylintcycle.first' >second.py
[proski@dell pylintcycle]$ pylint -r n --jobs=1 first.py second.py
************* Module pylintcycle.first
C:  1, 0: Missing module docstring (missing-docstring)
W:  1, 0: Unused import pylintcycle.second (unused-import)
************* Module pylintcycle.second
C:  1, 0: Missing module docstring (missing-docstring)
W:  1, 0: Unused import pylintcycle.first (unused-import)
R:  1, 0: Cyclic import (pylintcycle.first -> pylintcycle.second) (cyclic-import)
[proski@dell pylintcycle]$ pylint -r n --jobs=2 first.py second.py
************* Module pylintcycle.first
C:  1, 0: Missing module docstring (missing-docstring)
W:  1, 0: Unused import pylintcycle.second (unused-import)
************* Module pylintcycle.second
C:  1, 0: Missing module docstring (missing-docstring)
W:  1, 0: Unused import pylintcycle.first (unused-import)

@pylint-bot
Copy link
Author

Original comment by Pavel Roskin (BitBucket: pavel_roskin):


When running pylint -rn --jobs=2 pylint in an empty directory, following issue is reported:

:::text
************* Module pylint.lint
E:978,15: Instance of 'Values' has no 'persistent' member (no-member)

The single-job version does not detect that condition.

@pylint-bot
Copy link
Author

Original comment by Claudiu Popa (BitBucket: PCManticore, GitHub: @PCManticore):


Thank you for the detailed reports, Pavel. Will look soon.

@pylint-bot
Copy link
Author

Original comment by Claudiu Popa (BitBucket: PCManticore, GitHub: @PCManticore):


Regarding pylint -rn --jobs=2 pylint, this happens because there's no configuration files which disables the corresponding messages for pylint, such as the one from pylint/pylintrc. I can reproduce it with both the single-job version and with multiple jobs, it's actually expected.

@rdadolf
Copy link

rdadolf commented Jul 8, 2016

Late to the game and new to pylint, and I feel I'm running into this issue. I'm a little confused, though, since even very simple cases seem to break with -j2.

$ cat > lint_test.py <<EOF
def f(x):
 return x # 1 space
EOF

$ cat > pylintrc <<EOF
[MESSAGES CONTROL]
disable=all
enable=bad-indentation
EOF

$ pylint --version
pylint 1.6.1, 
astroid 1.4.7
Python 2.7.11 |Anaconda 2.3.0 (x86_64)| (default, Dec  6 2015, 18:57:58) 
[GCC 4.2.1 (Apple Inc. build 5577)]

$ pylint -rn lint_test.py 
************* Module lint_test
W:  2, 0: Bad indentation. Found 1 spaces, expected 4 (bad-indentation)

$ pylint -rn lint_test.py -j1
************* Module lint_test
W:  2, 0: Bad indentation. Found 1 spaces, expected 4 (bad-indentation)

$ pylint -rn lint_test.py -j2
<no output>

I don't see a way this can be expected behavior. Am I missing something? Am I using the tool fundamentally wrong?

@PCManticore
Copy link
Contributor

Hi @rdadolf

This bug is definitely irritating, I don't remember exactly why it got stuck, I should check again if it works.

For me your example works perfectly. Do you have any pylintrc in that directory or somewhere else, which could potentially interfere with Pylint? I tried the same configuration, on Windows. Did you try on another OS? Does it work with multiple jobs and can you test with multiple files?

@rdadolf
Copy link

rdadolf commented Jul 8, 2016

I sometimes forget that there are developers who use non-unix systems. It's possible that differences in process scheduling are at play here.

No other configuration files were in that directory, nor do I have a user-wide config file in my home directory.

The system above was OS X 10.11.3 running an Anaconda version of python (as shown in the --version output).

I just tried the same thing on an Ubuntu 15.10 machine which pylint was not previously installed. Used pip install pylint with no other actions, ran the same commands as above in a new scratch directory and I'm seeing the same behavior.

All values for N>1 with -jN seem to produce the same result (i.e., they fail to report anything).

Not sure what you mean by this last request, though:

Does it work with multiple jobs and can you test with multiple files?

The problem is identical for a multi-file module, if that's what you're asking. This setup:

$ ls -R
.:
lint_test  pylintrc

./lint_test:
__init__.py  lint_f.py  lint_g.py

$ cat lint_test/*.py
# __init__.py
from lint_f import f
from lint_g import g
# lint_f.py
def f(x):
 return x # 1 space
# lint_g.py
def g(x):
 return x # 1 space

I.e., a module with two source files and a simple __init__.py, also shows the same symptoms (same config file as before):

$ pylint -rn lint_test/
************* Module lint_test.lint_g
W:  3, 0: Bad indentation. Found 1 spaces, expected 4 (bad-indentation)
************* Module lint_test.lint_f
W:  3, 0: Bad indentation. Found 1 spaces, expected 4 (bad-indentation)
$ pylint -rn lint_test/ -j1
************* Module lint_test.lint_g
W:  3, 0: Bad indentation. Found 1 spaces, expected 4 (bad-indentation)
************* Module lint_test.lint_f
W:  3, 0: Bad indentation. Found 1 spaces, expected 4 (bad-indentation)
$ pylint -rn lint_test/ -j2
<no output>

@Pierre-Sassoulas
Copy link
Member

Adding a reproducer from a duplicate issue:

Running pylint -jobs=0 does not report about cyclic-import. This can be reproduced with a trivial example.
a.py:

import b

b.py:

import a

@mallniya
Copy link

I'm sorry to unearth this, but duplicate-code is still not shown with -j 2 with

pylint 2.14.4
astroid 2.11.7
Python 3.10.5 (main, Jun  6 2022, 18:49:26) [GCC 12.1.0]

@doublethefish
Copy link
Contributor

Which specific variant of duplicate code; in-file or across files?

@mallniya
Copy link

Only across files

@doublethefish
Copy link
Contributor

Can you share an example of two files that fail? I'll add it to the tests (which I am assuming are still working and do test -j2-10 on similar files).

@mallniya
Copy link

mallniya commented Aug 4, 2022

I guess there is nothing special about such files -- any similar lines in 2 files really:

$ cat bin/1.py
1
2
3
4
5
6

same for bin/2.py.

$ pylint bin/1.py bin/2.py 
...
bin/2.py:1:0: R0801: Similar lines in 2 files
==1:[0:6]
==2:[0:6]
1
2
3
4
5
6 (duplicate-code)

@seanbickle
Copy link

I experience the same behaviour (Pylint 2.15.0, Python 3.8.13). This behaviour seems to affect any check for which context is spread across files (cyclical imports, duplicate code).

At first I thought this was because input files were checked across different parallel workers, and therefore important context from a secondary file (e.g. duplicate code) was outside the context of the parallel worker. I don't think this is the case though; even when files are processed on the same worker (but jobs>1), the same behaviour is observed.

I think this has more to do with the different ways in which files are handled between serial and parallel execution (determined by the number of jobs). For example, for files foo and bar, I observed the similarity linter running twice for each file in parallel, but once for all files in serial. Of course if the similarity linter lints only foo, it will miss code duplicated in bar.

Another example not yet mentioned is invalid-name. test1.py:

'''
Docstring
'''

class Test:
    '''
    Test Class
    '''

    CLASS_ATTR = 5

test2.py:

'''
Docstring
'''
from test1 import Test

x = Test()
x.CLASS_ATTR = 3
~ % pylint --jobs=0 test1.py test2.py

--------------------------------------------------------------------
Your code has been rated at 10.00/10 (previous run: 10.00/10, +0.00)
~ % pylint --jobs=1 test1.py test2.py
************* Module test2
test2.py:7:0: C0103: Attribute name "CLASS_ATTR" doesn't conform to snake_case naming style (invalid-name)

-------------------------------------------------------------------
Your code has been rated at 8.00/10 (previous run: 10.00/10, -2.00)

@rpatterson
Copy link

I just lost a ton of time trying to debug why I was getting different results on CI than I was in any other environment even though the same Docker images were used both locally and on CI. Finally I found out it was because I had jobs = 0 to auto-detect the number of CPUs/cores and the CI runner had only one CPU. When I set jobs = 1 I got the same results in all environments. And it wasn't just "cyclical imports, duplicate code" checks, other errors were reported only in CI as well.

At the very least, I think the Parallel execution docs should include a big red flashing light to warn other developers who might get bitten by this.

@alexhayes
Copy link

I just lost a ton of time trying to debug why I was getting different results on CI than I was in any other environment even though the same Docker images were used both locally and on CI. Finally I found out it was because I had jobs = 0 to auto-detect the number of CPUs/cores and the CI runner had only one CPU. When I set jobs = 1 I got the same results in all environments. And it wasn't just "cyclical imports, duplicate code" checks, other errors were reported only in CI as well.

At the very least, I think the Parallel execution docs should include a big red flashing light to warn other developers who might get bitten by this.

I've also been battling really odd behaviour in CI when everything was running fine locally (in docker or otherwise). Changing jobs to 0 was what fixed it to me however (for some reason I had it set to 8 🤷)

@earonesty
Copy link

earonesty commented Mar 23, 2023

it's easy to repro:

  • have a massive code base, with lots of files
  • run pylint in parallel
  • get spurious errors

pylint with 1 core (jobs==1) never has this issue, but is way too slow for large code bases

@jacobtylerwalls
Copy link
Member

Thanks to all for the reproducers. Futuring the cyclic-import issue to #4171, and refocusing this issue on duplicate-code and closing it with #8757.

Some of the other comments in this thread are either no longer reproducible, or expected to be fixed by merged PRs:


📚 🚨 Please help us test the pylint and astroid alphas! 🚨 📚

If you can reproduce any further indeterminacy with --jobs on the main branches of pylint and astroid, please open a new issue with a minimal reproducer. Thanks!

@jacobtylerwalls
Copy link
Member

And the cyclic-import issue should now be fixed by #8807.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment