[ast branch] unicode literal fixes

[jpe]

BPO	1170272
Nosy	@brettcannon, @nascheme
Files	unicode.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brettcannon'
closed_at = <Date 2005-04-02.19:00:55.000>
created_at = <Date 2005-03-25.00:19:51.000>
labels = ['interpreter-core']
title = '[ast branch] unicode literal fixes'
updated_at = <Date 2005-04-02.19:00:55.000>
user = 'https://bugs.python.org/jpe'

bugs.python.org fields:

activity = <Date 2005-04-02.19:00:55.000>
actor = 'nascheme'
assignee = 'brett.cannon'
closed = True
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2005-03-25.00:19:51.000>
creator = 'jpe'
dependencies = []
files = ['6569']
hgrepos = []
issue_num = 1170272
keywords = ['patch']
message_count = 2.0
messages = ['48051', '48052']
nosy_count = 3.0
nosy_names = ['brett.cannon', 'nascheme', 'jpe']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1170272'
versions = []

[jpe]

Attached patch fixes test_eval by dealing with unicode
input correctly. I'm also compiling tokenizer.c
directly rather than tokenizer_pgen.c --
tokenizer_pgen.c simply defines PGEN and then includes
tokenizer.c, which disables some of the unicode
support. I don't think tokenizer_pgen.c is needed.

[nascheme]

Logged In: YES
user_id=35752

Commited. Thanks for the patch.