Flake8 doesn't seem to test tests

[mrocklin]

Does anyone know of a setting that would prevent flake8 from informing us about style errors on tests? I've noticed style issues on recent tests make it through the style checker. I can introduce tests locally and flake8 doesn't complain.

[jakirkham]

Do you have any ideas @sigmavirus24?

[sigmavirus24]

I don't see anything in your setup.cfg and I don't see anything in .travis.yml.

What versions of flake8 are y'all using and can you check the verbose output? It should help you debug why distributed.tests isn't being picked up.

[mrocklin]

mrocklin@carbon:~$ pip install flake8 --upgrade
cd Requirement already up-to-date: flake8 in ./Software/anaconda/lib/python3.6/site-packages
Requirement already up-to-date: mccabe<0.7.0,>=0.6.0 in ./Software/anaconda/lib/python3.6/site-packages/mccabe-0.6.1-py3.6.egg (from flake8)
Requirement already up-to-date: pycodestyle<2.4.0,>=2.0.0 in ./Software/anaconda/lib/python3.6/site-packages (from flake8)
Requirement already up-to-date: pyflakes<1.7.0,>=1.5.0 in ./Software/anaconda/lib/python3.6/site-packages (from flake8)
wmrocklin@carbon:~$ cd workspace/distributed
mrocklin@carbon:~/workspace/distributed$ flake8 --verbose distributed
flake8.plugins.manager    MainProcess    370 INFO     Loading entry-points for "flake8.extension".
flake8.plugins.manager    MainProcess    529 INFO     Loading entry-points for "flake8.listen".
flake8.plugins.manager    MainProcess    529 INFO     Loading entry-points for "flake8.report".
flake8.plugins.manager    MainProcess    530 INFO     Loading plugin "F" from entry-point.
flake8.plugins.manager    MainProcess    552 INFO     Loading plugin "pycodestyle.ambiguous_identifier" from entry-point.
flake8.plugins.manager    MainProcess    561 INFO     Loading plugin "pycodestyle.bare_except" from entry-point.
flake8.plugins.manager    MainProcess    561 INFO     Loading plugin "pycodestyle.blank_lines" from entry-point.
flake8.plugins.manager    MainProcess    561 INFO     Loading plugin "pycodestyle.break_around_binary_operator" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.comparison_negative" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.comparison_to_singleton" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.comparison_type" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.compound_statements" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.continued_indentation" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.explicit_line_join" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.extraneous_whitespace" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.imports_on_separate_lines" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.indentation" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.maximum_line_length" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.missing_whitespace" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.missing_whitespace_after_import_keyword" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.missing_whitespace_around_operator" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.module_imports_on_top_of_file" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.python_3000_backticks" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.python_3000_has_key" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.python_3000_not_equal" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.python_3000_raise_comma" from entry-point.
flake8.plugins.manager    MainProcess    562 INFO     Loading plugin "pycodestyle.tabs_obsolete" from entry-point.
flake8.plugins.manager    MainProcess    563 INFO     Loading plugin "pycodestyle.tabs_or_spaces" from entry-point.
flake8.plugins.manager    MainProcess    563 INFO     Loading plugin "pycodestyle.trailing_blank_lines" from entry-point.
flake8.plugins.manager    MainProcess    563 INFO     Loading plugin "pycodestyle.trailing_whitespace" from entry-point.
flake8.plugins.manager    MainProcess    563 INFO     Loading plugin "pycodestyle.whitespace_around_comma" from entry-point.
flake8.plugins.manager    MainProcess    563 INFO     Loading plugin "pycodestyle.whitespace_around_keywords" from entry-point.
flake8.plugins.manager    MainProcess    563 INFO     Loading plugin "pycodestyle.whitespace_around_named_parameter_equals" from entry-point.
flake8.plugins.manager    MainProcess    563 INFO     Loading plugin "pycodestyle.whitespace_around_operator" from entry-point.
flake8.plugins.manager    MainProcess    563 INFO     Loading plugin "pycodestyle.whitespace_before_comment" from entry-point.
flake8.plugins.manager    MainProcess    563 INFO     Loading plugin "pycodestyle.whitespace_before_parameters" from entry-point.
flake8.plugins.manager    MainProcess    563 INFO     Loading plugin "C90" from entry-point.
flake8.plugins.manager    MainProcess    564 INFO     Loading plugin "default" from entry-point.
flake8.plugins.manager    MainProcess    566 INFO     Loading plugin "pylint" from entry-point.
flake8.plugins.manager    MainProcess    566 INFO     Loading plugin "quiet-filename" from entry-point.
flake8.plugins.manager    MainProcess    566 INFO     Loading plugin "quiet-nothing" from entry-point.
flake8.checker            MainProcess    586 INFO     Making checkers
flake8.checker            MainProcess    762 INFO     Checking 107 files
flake8.main.application   MainProcess   3796 INFO     Finished running
flake8.main.application   MainProcess   3796 INFO     Reporting errors
flake8.main.application   MainProcess   3801 INFO     Found a total of 135 violations and reported 0
mrocklin@carbon:~/workspace/distributed$ flake8 --version
3.5.0 (mccabe: 0.6.1, pycodestyle: 2.3.1, pyflakes: 1.6.0) CPython 3.6.0 on Linux

[sigmavirus24]

@mrocklin try flake8 -vv distributed. That will give you a lot more information but should tell you whether or not the test files are being checked at all.

[mrocklin]

Yup. That's 180MB of data. Any suggestions of what to look for?

…

On Sat, Dec 9, 2017 at 11:38 AM, Ian Stapleton Cordasco < ***@***.***> wrote: @mrocklin <https://github.com/mrocklin> try flake8 -vv distributed. That will give you *a lot* more information but should tell you whether or not the test files are being checked at all. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1612 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszDNPsgvgAgKSZw8Wqyy0PFR6oNDcks5s-rd2gaJpZM4Q08ha> .

[sigmavirus24]

Look for lines like has been excluded. You can do flake8 -vv distributed | grep "has been excluded". That should tell you if someone or something has exclude set to ignore test files.

[mrocklin]

That line does not occur around the testing files.

flake8.checker            MainProcess    621 DEBUG    "/home/mrocklin/workspace/distributed/distributed/tests/test_scheduler.py" has not been excluded
flake8.checker            MainProcess    622 DEBUG    "/home/mrocklin/workspace/distributed/test_scheduler.py" has not been excluded

[sigmavirus24]

Cool. So Flake8 should be running against those files. And it doesn't appear that there are ERRORs in flake8 -v output.

So let me get this straight, if you run flake8 distributes/tests/test_schedule.py that shows errors that you otherwise don't see?

[mrocklin]

No, even running flake8 against specific files it will not show errors. However I can introduce style errors into those files directly and even when running flake8 against them those errors will not show up.

Looking into this further, I think that this is due to a misunderstanding on my part of the use of the #flake8: noqa directive. Previously we wrote this next to a few import statements

from foo import bar # flake8: noqa

def test_badly_styled_test(bar):
    x = 1+2    - 3

My understanding was that the #flake8: noqa directive would only apply to the import line, but it seems instead to apply to the entire file.

[sigmavirus24]

Yes it does apply to the whole file. You want to use # noqa: F401 or whatever the error is there.

[mrocklin]

Is there a way to avoid checks on a particular line? I value flake8 checking that particular error in most cases.

[sigmavirus24]

@mrocklin You can use # noqa: <violation>[,<violation>] to ignore a violation on a particular line. We have this documented here: http://flake8.pycqa.org/en/latest/user/violations.html#in-line-ignoring-errors

[mrocklin]

This has been resolved. Thanks everyone.