Addressing PEP8 by including pytest-flake8

[noviluni]

As mentioned in the Scrapy Coding Style section in the docs (https://docs.scrapy.org/en/master/contributing.html#coding-style) :

“Unless otherwise specified, follow PEP 8.”.

However, it’s known that the current codebase has a lot of parts that don’t respect PEP8 guidelines (#2144).

I know that it’s difficult to address that, as it involves a lot of code to be reviewed but, however, the more time we skip that task the harder it becomes.

Two obvious things that we could do are:

• Fixing the current code
• Adding a pipeline to check pep8

Fixing the current code can’t be done in just one commit, but instead of fixing pep8 in specific project parts (example: #2207, #2429), it could better and easier to review if we fix rules one by one (for example: "Unused imports", "Missing blank lines after class or function definition."...).

My proposal to avoid to write a hard to maintain pipeline is to include the pytest-flake8 package that executes flake8 when executing pytest (by adding the --flake8 flag) and raises an error when the check fails.

If we start excluding all the rules then we can just fix one by one from more obvious rules to more discussed rules just excluding that rule from the ignored rules list. In that way we ensure that new code respects the previous agreed flake8/pep8 rules and on the other hand we can discuss every rule if we want to follow it or not (example: #3697).

I will add a PR to illustrate this idea.

[noviluni]

What do you think?? I’ve been evaluating other possible solutions and I’ve seen that scikit-learn uses a pipeline to check flake8 against the new/diff lines with a bash script https://github.com/scikit-learn/scikit-learn/blob/master/build_tools/circle/flake8_diff.sh but it seems really hard to mantain.

[Gallaecio]

Maybe we could have a separate tox environment for these analysis? (#3727 could also be run there)

[noviluni]

Hi @Gallaecio . Sounds good, I will try to update my PR when I have time.

[noviluni]

Hi @Gallaecio I've updated the PR (#3945) with a new tox environment and a its travis pipeline and it seems that worked. Feel free to review it and give your feedback :)

[noviluni]

The PR (#3945) has been already merged so I proceed closing this. Thank you @Gallaecio 🎉 😄