Spawning child-process when using multiprocessing is very slow

[Pierre-Sassoulas]

Bug description

See: #6965 (comment)

I've noticed that spawning the child processes is extremely slow - about 1-2 processes per second. So, 60-way parallelism generally means that there is about a 30 s delay while things spin up, then almost no time spent doing work. This slow startup time was happening despite my CPUs being about 70% idle. This slow startup time presumably explains the bug I saw about how ineffectual multiprocessing was for pylint. You would need a huge batch of files to justify doing significant multi-processing.

I don't know whether the slow startup is a bug in multiprocessing or in pylint. I just measured some of our presubmits and 60-way parallelism more than doubles the time that they take, from ~30 to ~70 s.

Possible solution in #6965 (comment)

We would need a complete rewrite of the parallel code. We currently spin up a new PyLinter class for every job. That is taking way too long (probably). But I'm not sure what the best approach is to create a PyLinterLite...

I personally think that a refactor of PyLinter will be required, and we'd have to classify checkers to know if they can benefit from multiprocessing or not. duplicate-code or cyclic import won't for example as they need information on the imports of a file. Some check are are file based like unused-private-member (the scope is a single class) or while-used (it just has to check if a while node exists) and can benefit from multiprocessing if done at the right time.

Configuration

We should use a full configuration for this with a lot to parse, as we're probably parsing the configuration in each forks and this would make it apparent.

Command used

``lint.Run(['--jobs', '42'] + argv)``

Expected behavior

Run time decrease with more core (when there is more files to lint than cores available).

Pylint version

2.14.2

[DanielNoord]

Duplicate of #2525 😄