W191: missed when file ends without a newline

[jaraco]

I'm working on a project that enables flake8 only for testing for tabs for indentation (they really don't want tabs).

In working on a minimal test to demonstrate the effectiveness of the check, I created a small Python file with tabs for indentation, then ran flake8 on it, only to find that it passed the check:

$ python -c "import pathlib; pathlib.Path('tabbed.py').write_text('def foo():\\n\\tpass')"
$ pip-run -q pycodestyle -- -m pycodestyle --select=W191 tabbed.py
$

But add a newline to the file and it fails the check:

$ python -c "import pathlib; pathlib.Path('tabbed.py').write_text('def foo():\\n\\tpass\\n')"
$ pip-run -q pycodestyle -- -m pycodestyle --select=W191 tabbed.py
tabbed.py:2:1: W191 indentation contains tabs

Both files contain tabs, but the former manages not to catch the error.

[asottile]

interesting!

comparing the tokenization I'm not quite sure why this is happening:

$ python3 -m tokenize f.py 
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,3:            NAME           'def'          
1,4-1,5:            NAME           'f'            
1,5-1,6:            OP             '('            
1,6-1,7:            OP             ')'            
1,7-1,8:            OP             ':'            
1,8-1,9:            NEWLINE        '\n'           
2,0-2,1:            INDENT         '\t'           
2,1-2,5:            NAME           'pass'         
2,5-2,6:            NEWLINE        ''             
3,0-3,0:            DEDENT         ''             
3,0-3,0:            ENDMARKER      ''          

$ python3 -m tokenize f2.py 
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,3:            NAME           'def'          
1,4-1,5:            NAME           'f'            
1,5-1,6:            OP             '('            
1,6-1,7:            OP             ')'            
1,7-1,8:            OP             ':'            
1,8-1,9:            NEWLINE        '\n'           
2,0-2,1:            INDENT         '\t'           
2,1-2,5:            NAME           'pass'         
2,5-2,6:            NEWLINE        '\n'           
3,0-3,0:            DEDENT         ''             
3,0-3,0:            ENDMARKER      ''    

(diff so it's easier to read)

$ diff -u <(python3 -m tokenize f.py) <(python3 -m tokenize f2.py)
--- /dev/fd/63	2020-09-11 12:26:08.174802023 -0700
+++ /dev/fd/62	2020-09-11 12:26:08.174802023 -0700
@@ -7,6 +7,6 @@
 1,8-1,9:            NEWLINE        '\n'           
 2,0-2,1:            INDENT         '\t'           
 2,1-2,5:            NAME           'pass'         
-2,5-2,6:            NEWLINE        ''             
+2,5-2,6:            NEWLINE        '\n'           
 3,0-3,0:            DEDENT         ''             
 3,0-3,0:            ENDMARKER      ''     

[asottile]

looking at this, it appears that all physical line checks do not validate the last line of a file which does not end in a newline!

for example, the trailing whitespace check:

$ echo -en 'x = 1     ' > t.py
$ python3 -m pycodestyle t.py 
t.py:1:11: W292 no newline at end of file

let's see how simple this is to fix 🤔

[asottile]

check out #961 and see if it works for you!