[BUG] Prospector endless cycle. Check never ends

[ObjatieGroba]

Describe the bug

After upgrading from 1.7 to 1.8 with pylint 2.15.8

To Reproduce Steps to reproduce the behavior:

1. Create any empty py file
2. Create .pylintrc file with following text:

[MASTER]
jobs=2

3. prospector
4. ... Never ends

Expected behavior
Success check

Screenshots
py-spy process analyzing:

Environment (please complete the following information):

• OS: Ubuntu 22.04
• Tool pylint
• Prospector version 1.8.0-1.8.2
• Python version 3.10

Additional context

If jobs is 1 - everything is OK.
Does pylint fork makes prospector to be unstoppable?

[jzskca]

I see this as well. Upgrading to pylint 2.15.9 fixes the infinite loop and results in a backtrace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prospector/run.py", line 87, in execute
    messages += tool.run(found_files)
  File "/usr/local/lib/python3.9/site-packages/prospector/tools/pylint/__init__.py", line 239, in run
    self._linter.check(self._args)
  File "/usr/local/lib/python3.9/site-packages/pylint/lint/pylinter.py", line 674, in check
    check_parallel(
  File "/usr/local/lib/python3.9/site-packages/pylint/lint/parallel.py", line 165, in check_parallel
    ) in executor.map(_worker_check_single_file, files):
  File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 726, in map
    results = super().map(partial(_process_chunk, fn),
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 598, in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 598, in <listcomp>
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
  File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 681, in submit
    raise BrokenProcessPool(self._broken)
concurrent.futures.process.BrokenProcessPool: A child process terminated abruptly, the process pool is not usable anymore

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/prospector", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/prospector/run.py", line 199, in main
    prospector.execute()
  File "/usr/local/lib/python3.9/site-packages/prospector/run.py", line 105, in execute
    raise FatalProspectorException(f"Tool {toolname} failed to run.") from ex
prospector.exceptions.FatalProspectorException: Tool pylint failed to run.

Downgrading pylint to 2.12.2 allows prospector to run correctly again. 2.13.0 and higher, up to 2.15.8, results in the infinite loop behaviour. pylint 2.15.9 works correctly when run directly.

# pylint --version
pylint 2.15.9
astroid 2.12.13
Python 3.9.10 (main, Mar  2 2022, 04:23:34) 
[GCC 10.2.1 20210110]
# prospector --version
prospector 1.8.3

Edit: The above happens with jobs=0. With jobs=1 everything works correctly.

[Pierre-Sassoulas]

Should we close as it's an upstream bug in pylint ?

[ObjatieGroba]

Should we close as it's an upstream bug in pylint ?

I do not think so. Pylint manual running works well. Prospector should fix that because it runs pylint wrong way

[koniiiik]

This is one of the possible symptoms of the issue that was fixed in #577.