PEP 414 installation hook fails with an AssertionError

[vsajip]

BPO	14713
Nosy	@vsajip, @mitsuhiko
Superseder	bpo-9974: tokenizer.untokenize not invariant with line continuations

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-04-23.20:47:04.656>
created_at = <Date 2012-05-03.15:42:20.922>
labels = ['type-bug']
title = 'PEP 414 installation hook fails with an AssertionError'
updated_at = <Date 2016-04-23.20:47:04.656>
user = 'https://github.com/vsajip'

bugs.python.org fields:

activity = <Date 2016-04-23.20:47:04.656>
actor = 'berker.peksag'
assignee = 'none'
closed = True
closed_date = <Date 2016-04-23.20:47:04.656>
closer = 'berker.peksag'
components = []
creation = <Date 2012-05-03.15:42:20.922>
creator = 'vinay.sajip'
dependencies = []
files = []
hgrepos = []
issue_num = 14713
keywords = []
message_count = 2.0
messages = ['159864', '174720']
nosy_count = 2.0
nosy_names = ['vinay.sajip', 'aronacher']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '9974'
type = 'behavior'
url = 'https://bugs.python.org/issue14713'
versions = ['Python 3.3']

[vsajip]

I'm not sure if I've done something wrong, but I get an AssertionError when trying to run the tokenizer on a Python file from the Django source. The gist at https://gist.github.com/1977558 has the files concerned:

1. test_tokenize.py - the script which fails
2. tokenize_example.py - the file being tokenized and untokenized. This is from the Django source: django/extras/csrf_migration_helper.py
3. tokenizer.py - your tokenize module, I renamed it because I was working in /tmp and didn't want to import the Python 3.2 stdlib's tokenize.py
4. The test output shows that the tokenize_example module imports OK in Python 2.7.2, but running the test_tokenize script on it with Python3.2 fails with an AssertionError.

I did some more testing, there are 131 failures in the Django source tree (all look like the same AssertionError).

N.B. I posted this to your GitHub repo where you published the hook.

[akuchling]

Duplicate of bpo-9974. The tokenizer is choking on lines that end in '\'. So even this can't be parsed:

a = 1 + \
   2