duplicate-code is not detected when using more than one core

[transreductionist]

Description of Bug

Duplicate code works in 2.13.9 and does not work in 2.14.0.

When using more than one core duplicate-code is not detected.

The machine pylint runs on has 4 cores. Running with one core:

pylint --rcfile pylint_test/rc0.cfg j 1 pylint_test0

Pylint identifies all errors as expected:

• line-too-long
• duplicate code

When running with 2 of 4 cores:

pylint --rcfile pylint_test/rc0.cfg j 2 pylint_test0

pylint does not identify duplicate-code, and only finds:

• line-too-long

Configuration Parameters

• max-line-length
• duplicate-code

How to Reproduce

Here is how to reproduce the different errors depending on number of cores selected.

I created 2 files in a test directory called: pylint_tests. The files are identical (see below). The parameter min-similarity-lines is left as the default value: 4. The max-line-length is set to 10 so there is a second message that should be raised. This is needed to see the difference in pylint output depending on the cores selected.

Within the pylint_test directory the pylint config file was built automatically. The only change made to the file, as mentioned above, is that the max-line-length was changed from the default to 10 lines.

pylint --generate-rcfile > rc0.cfg

The 2 identical files that are in the pylint_test directory are pylint_test01.py and ``pylint_test02.py. Here is pylint_test01.py`:

"""Docstring"""

def pylint_test01:
    """Docstring"""
    
    a_squared00 = 1
    a_squared01 = a_squared00
    a_squared02 = a_squared01
    if a_squared02:
        pass

Pylint Output

one_and_three_core_pylint_messages.txt

Configuration

Pylint Configuration File


[FORMAT]

# Maximum number of characters on a single line.
max-line-length=10. 


[SIMILARITIES]

# Ignore comments when computing similarities.
ignore-comments=yes

# Ignore docstrings when computing similarities.
ignore-docstrings=yes

# Ignore imports when computing similarities.
ignore-imports=yes

# Signatures are removed from the similarity computation
ignore-signatures=yes

# Minimum lines number of a similarity.
min-similarity-lines=4

Command used

In the parent directory for pylint_test the following command is run with either (N) 1 or 2 cores. The machine has 4 cores.


pylint --rcfile pylint_test/rc0.cfg j N pylint_test0

Pylint output

See Bug Description section of form for the pylint output text file.

Expected behavior

It is expected that when running pylint with 2 out of 4 cores that the duplicate-code is reported.

Pylint version

pylint 2.16.1
astroid 2.14.1
python 3.8.3

OS / Environment

Microsoft Windows
Version 20H2 (OSBuild 19042.2486)
Windows 10 Enterprise

Git version 2.8.2.windows.1

Additional dependencies

No response

[jacobtylerwalls]

Thanks for the well-written report. Unfortunately, if it worked in 2.13.9, I'd say you were getting lucky more than anything else. The fact that --jobs produces incorrect output is IMO the biggest open wart in pylint's codebase at the moment. If it's not addressed in 3.0, we should enumerate the problems with it in the docs and caution people against using it. Closing as a duplicate of #374 (open since 2014).