check_parallel| Adds new test suite for multiple workers/check_parallel

[doublethefish]

Steps

• (see PR Fix #3314 duplicate code error only shows up with pylint jobs 1 #3458) Add yourself to CONTRIBUTORS if you are a new contributor.
• (see PR Fix #3314 duplicate code error only shows up with pylint jobs 1 #3458) Add a ChangeLog entry describing what your PR does.
• If it's a new feature or an important bug fix, add a What's New entry in doc/whatsnew/<current release.rst>.
• Write a good description on what the PR does.

Description

These regression tests put check_parallel() and related functions under test, ensuring that the functionality remains consistent ahead of any bug fixes and performance work we may do.

This makes it easier to make larger changes here later, for example performance work (see PR #3458, and issue #2525).

The test uses the stats generated by a lint-run to regression-test that the correct number of operations have been performed on the expected number of files irrespective of the number of workers being used.

Really this work is a pre-cursor to changing how check_parallel functions (PR #3458). So we check:

• Fewer workers than files
  • We currently assign individual files to a worker, if we change this it is good to ensure consistent behaviour.
• Fewer files than workers
  • We want to ensure that no more work than required is done and we saturate as many workers as possible to get the work done quicker (which will happen once we solve the wall-clock issues mentioned in PR #)
• 1, 2 or 3 checkers
  • We want to ensure that each checker is run just once per file, irrespective of the number of workers or files.
  • Later, if we add more Checker-types that change how we work across workers we will need to ensure that each Checker-type works consistently for all edge cases e,g, see PR Fix #3314 duplicate code error only shows up with pylint jobs 1 #3458 where we add a map/reduce Checker-mixin

Type of Changes

	Type
✓	🔨 Refactoring

Related Issue

[coveralls]

Coverage increased (+0.08%) to 90.534% when pulling 06b1e03 on doublethefish:feature/test_check_parallel into 22fe050 on PyCQA:master.

[coveralls]

Coverage increased (+0.08%) to 90.534% when pulling 06b1e03 on doublethefish:feature/test_check_parallel into 22fe050 on PyCQA:master.

[coveralls]

Coverage increased (+0.2%) to 90.782% when pulling c85297c on doublethefish:feature/test_check_parallel into 707fc46 on PyCQA:master.

[doublethefish]

This is currently broken by #3516. @Pierre-Sassoulas.

[Pierre-Sassoulas]

Could you try adding missing import in pylint/lint/__init__.py ? As there is no changes in lint.py I think this should be enough.

[doublethefish]

Unfortunately that didn't work.

Part of the reason for it is that the new file-module (check_parallel.py) is hidden by the function import: from pylint.lint.check_parallel import check_parallel.

The above means we can't directly access properties in the check_parallel module in a read/write manner and everything else feels like a workaround.

I am going to rename check_parallel.py -> parallel.py, its more descriptive. It looks like that the only file-module with that problem after the refactor.

[PCManticore]

Hey @doublethefish Thanks for the PR, this has been a great one. I left a bunch of comments that we should address before pulling this in.

[Pierre-Sassoulas]

It seems all of @PCManticore comments were addressed, I think we can merge this. The only thing is that with time a conflict appeared and we can't rebase automatically anymore. Could you rebase on the latest master @doublethefish ?

[brycepg]

I'd be happy to fix the conflict if you aren't available @doublethefish Will check back later

[brycepg]

Added this in another PR, thank for the added tests @doublethefish

[doublethefish]

Once again, thanks for sorting this. It's been a difficult 6 weeks.