Refactor file checking for a simpler parallel API (#3016)

This change prepares the code for enabling Prospector to take advantage
of running PyLint parallel.

Iterating files is moved into generator (_iterate_file_descrs) so that
parallel checking can use the same implementation (_check_file) just
by providing different kind of generator that reads the files from parent
process.

The refactoring removes code duplication that existed in PyLinter._do_check
method; checking module content from stdin had identical implementation to
checking content from a source file.

Made PyLinter.expand_files a private method.

The previous implementation of parallel linting created new PyLinter
objects in the worker (child) process causing failure when running under
Prospector because Prospector uses a custom PyLinter class
(a class inherited from PyLinter)
and PyLint naturally just creates PyLinter object. This caused linting to
fail because there is options for Prospector's IndentChecker which was not
created in the worker process.

The new implementation passes the original PyLinter object into workers
when the workers are created. See https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

Note that as Windows uses spawn method by default, PyLinter object (and
its) members need to be pickleable from now on with the exception being
PyLinter.reporter which is not passed to child processes.

The performance has remained about the same based on quick tests done with
Django project containing about 30 000 lines of code; with the old
implementation linting took 26-28 seconds with 8 jobs on quad core i7 and
24-27 seconds with the new implementation.

CONTRIBUTORS.txt

@@ -339,3 +339,4 @@ contributors:
 
 * laike9m: contributor
 
+* Janne Rönkkö: contributor

ChangeLog

@@ -31,11 +31,21 @@ Release date: TBA
 
   Closes #2729
 
+* Allow parallel linting when run under Prospector
+
 
 What's New in Pylint 2.4.3?
 ===========================
 
-Release date: TBA
+  Pass the actual PyLinter object to sub processes to allow using custom
+  PyLinter classes.
+
+  PyLinter object (and all its members except reporter) needs to support
+  pickling so the PyLinter object can be passed to worker processes.
+
+* Refactor file checking
+
+  Remove code duplication from file checking.
 
 * Fix an issue with ``unnecessary-comprehension`` in comprehensions with additional repacking of elements.
 

pylint/checkers/classes.py

@@ -634,11 +634,15 @@ def _has_same_layout_slots(slots, assigned_value):
 }
 
 
+def _scope_default():
+    return collections.defaultdict(list)
+
+
 class ScopeAccessMap:
     """Store the accessed variables per scope."""
 
     def __init__(self):
-        self._scopes = collections.defaultdict(lambda: collections.defaultdict(list))
+        self._scopes = collections.defaultdict(_scope_default)
 
     def set_accessed(self, node):
         """Set the given node as accessed."""

pylint/config.py

@@ -37,6 +37,7 @@
 import configparser
 import contextlib
 import copy
+import functools
 import io
 import optparse
 import os
@@ -719,10 +720,8 @@ def read_config_file(self, config_file=None, verbose=None):
             opt = "-".join(["long"] * helplevel) + "-help"
             if opt in self._all_options:
                 break  # already processed
-            # pylint: disable=unused-argument
-            def helpfunc(option, opt, val, p, level=helplevel):
-                print(self.help(level))
-                sys.exit(0)
+
+            helpfunc = functools.partial(self.helpfunc, level=helplevel)
 
             helpmsg = "%s verbose help." % " ".join(["more"] * helplevel)
             optdict = {"action": "callback", "callback": helpfunc, "help": helpmsg}
@@ -830,6 +829,10 @@ def help(self, level=0):
         with _patch_optparse():
             return self.cmdline_parser.format_help()
 
+    def helpfunc(self, option, opt, val, p, level):  # pylint: disable=unused-argument
+        print(self.help(level))
+        sys.exit(0)
+
 
 class OptionsProviderMixIn:
     """Mixin to provide options to an OptionsManager"""