-
Notifications
You must be signed in to change notification settings - Fork 172
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix pylint parallel execution #577
Fix pylint parallel execution #577
Conversation
One minor question but otherwise LGTM, thanks for investigating and finding this! Another question, is the |
Oh, yeah, that's rather arbitrary, I created multiple files right off the bat, because I wasn't sure if parallel execution would fully trigger without them, but I might be able to clean it up. |
It's not an issue to leave it, it just seemed an anomaly so wanted to check! |
1e4a180
to
e577824
Compare
Turns out it's not necessary, so in the interest of avoiding any potential confusion, I cleaned it up. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think upgrading to python 3.7.2 is not that big of a deal, some tool like flake8 already dropped python 3.7. And the EOL for the latest python 3.7 is very soon anyway.
Description
This patch prevents closing over the entire
ProspectorConfig
in the filter generated bymake_exclusion_filter
to avoid pickling the entire object when pylint forks subprocesses for its parallel execution.Related Issue
#573
Motivation and Context
Full context explained in #573 (comment)
This branch also includes a slight bump of the lower bound on Python in pyproject.toml. The reason is that allowing Python 3.7.1 locks pylint to versions below 2.14, and pylint 2.13 has an even more catastrophic failure mode than 2.15 – instead of crashing, it livelocks in an endless loop of spawning new subprocesses.
Lifting the Python lower bound to 3.7.2 allows poetry to upgrade pylint to 2.15, which handles the failure state in a slightly more graceful way.
How Has This Been Tested?
Ran pytest on Python 3.10. Also makes it possible for prospector to run successfully on our code base at $DAYJOB.
Types of changes
Checklist: