Support \u and \U escapes in regexes

[birkenfeld]

BPO	3665
Nosy	@birkenfeld, @atsuoishimoto, @pitrou, @ezio-melotti, @merwok, @serhiy-storchaka
Files	re_unicode_escapes.diff 3665.patch re_unicode_escapes.diff: Regenerate georg.brandl's patch for review 3665.patch: Regenerate ishimoto's patch for review re_unicode_escapes-2.patch: + PEP 393, + cleanup, + tests re_unicode_escapes-3.patch: + byte patterns, + tests, + docs

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/pitrou'
closed_at = <Date 2012-06-23.11:33:41.166>
created_at = <Date 2008-08-24.20:33:51.445>
labels = ['expert-regex', 'type-feature', 'library', 'expert-unicode']
title = 'Support \\u and \\U escapes in regexes'
updated_at = <Date 2012-06-23.11:48:13.258>
user = 'https://github.com/birkenfeld'

bugs.python.org fields:

activity = <Date 2012-06-23.11:48:13.258>
actor = 'serhiy.storchaka'
assignee = 'pitrou'
closed = True
closed_date = <Date 2012-06-23.11:33:41.166>
closer = 'pitrou'
components = ['Library (Lib)', 'Regular Expressions', 'Unicode']
creation = <Date 2008-08-24.20:33:51.445>
creator = 'georg.brandl'
dependencies = []
files = ['11235', '17939', '25783', '25784', '26035', '26040']
hgrepos = []
issue_num = 3665
keywords = ['patch', 'needs review']
message_count = 14.0
messages = ['71861', '71864', '71865', '71868', '109961', '138219', '162052', '162830', '163065', '163094', '163580', '163584', '163585', '163590']
nosy_count = 9.0
nosy_names = ['georg.brandl', 'ishimoto', 'pitrou', 'timehorse', 'ezio.melotti', 'eric.araujo', 'mrabarnett', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'critical'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue3665'
versions = ['Python 3.3']

[birkenfeld]

Since \u and \U aren't interpolated in raw strings anymore, the re
module should support those escapes in addition to the \x and octal ones
it already does. Attached patch.

[pitrou]

• Check that it also works for chars > 0xFFFF (even in UCS2 builds, at
  least when the chars are not part of [character range])
• What does happen with e.g. [\U00010000-\U00010001] on an UCS build?

[pitrou]

(in the last sentence, I meant UCS2. Sorry)

[birkenfeld]

These concerns indeed must be handled: On narrow unicode builds, chars >
0xffff must be converted to surrogates. In ranges, they should raise an
error.

Additionally, this should at least raise an error too:

>>> re.compile("[\U00100000]").match("\U00100000").group()
'\udbc0'

[atsuoishimoto]

Here's an updated patch for py3k branch.
As per Georg's comment, I added to check codepoint in the character
ranges, conversion to the surrogate pairs. I also added check to raise
exception if codepoint > 0x10ffff.
I with to English speakers to fix error messages in the patch.

[merwok]

FYI,
+ raise error("bogus escape: %s" % repr(escape))

can be written simply as

+ raise error("bogus escape: %r" % escape)

[serhiy-storchaka]

I don't think it is worth to target it for 2.7 and 3.2 (it's new feature, not bugfix), but for 3.3 it will be very useful.

Since PEP-393 conversion to the surrogate pairs is no longer relevant.

[serhiy-storchaka]

Georg, Atsuo, how are you?

[serhiy-storchaka]

Here is updated (in conforming with PEP-393) patch. In additional octal and hexadecimal escaping cleared, illegal error message for hexadecimal escaping fixed. Added new tests for octal and hexadecimal escaping.

[serhiy-storchaka]

I forgot about byte patterns. Here is an updated patch.

[serhiy-storchaka]

Any chance to commit the patch today and to get this feature in Python 3.3?

[python-dev]

New changeset b1dbd8827e79 by Antoine Pitrou in branch 'default':
Issue bpo-3665: \u and \U escapes are now supported in unicode regular expressions.
http://hg.python.org/cpython/rev/b1dbd8827e79

[pitrou]

Any chance to commit the patch today and to get this feature in Python
3.3?

Thanks for reminding us! It's now in 3.3.

[serhiy-storchaka]

Thank you for the quick response.