Commits on Oct 17, 2019

1. Refactor file checking for a simpler parallel API …

   This change prepares the code for enabling Prospector to take advantage
   of running PyLint parallel.
   
   Iterating files is moved into generator (_iterate_file_descrs) so that
   parallel checking can use the same implementation (_check_file) just
   by providing different kind of generator that reads the files from parent
   process.
   
   The refactoring removes code duplication that existed in PyLinter._do_check
   method; checking module content from stdin had identical implementation to
   checking content from a source file.
   
   Made PyLinter.expand_files a private method.
   
   The previous implementation of parallel linting created new PyLinter
   objects in the worker (child) process causing failure when running under
   Prospector because Prospector uses a custom PyLinter class
   (a class inherited from PyLinter)
   and PyLint naturally just creates PyLinter object. This caused linting to
   fail because there is options for Prospector's IndentChecker which was not
   created in the worker process.
   
   The new implementation passes the original PyLinter object into workers
   when the workers are created. See https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
   
   Note that as Windows uses spawn method by default, PyLinter object (and
   its) members need to be pickleable from now on with the exception being
   PyLinter.reporter which is not passed to child processes.
   
   The performance has remained about the same based on quick tests done with
   Django project containing about 30 000 lines of code; with the old
   implementation linting took 26-28 seconds with 8 jobs on quad core i7 and
   24-27 seconds with the new implementation.

   

   janneronkko committed Oct 17, 2019
   Configuration menu

   • View commit details
   • Copy full SHA for 21dc875
   • Browse repository at this point

   Copy the full SHA

   21dc875 View commit details

   Browse the repository at this point in the history