Python tokenize

ENCODING

PEP 263 (Python >= 2.3) defines a comment format that declares the source coding of a source file. It must

1. Be on the 1st or 2nd line of the file
2. Match ^[ \t\f]*#.*?coding[:=][ \t]*([-_.a-zA-Z0-9]+)

Given

# coding: utf-8
pass

Python 2

The tokenize module emits only a COMMENT token

➜  mitogen git:(master) ✗ python2 -m tokenize coding.py                                                    
1,0-1,15:       COMMENT '# coding: utf-8'
1,15-1,16:      NL      '\n'
2,0-2,4:        NAME    'pass'
2,4-2,5:        NEWLINE '\n'
3,0-3,0:        ENDMARKER       ''

Python 3

The tokenize module emits both an ENCODING and a COMMENT token

➜  mitogen git:(master) ✗ python3 -m tokenize coding.py
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,15:           COMMENT        '# coding: utf-8'
1,15-1,16:          NL             '\n'           
2,0-2,4:            NAME           'pass'         
2,4-2,5:            NEWLINE        '\n'           
3,0-3,0:            ENDMARKER      ''