Underscores in numeric literals not supported in lib2to3.

[nevsan]

BPO	29869
Nosy	@birkenfeld, @serhiy-storchaka, @Mariatta, @nevsan
PRs	bpo-29869: Allow underscores in numeric literals in lib2to3. #752 Revert bpo-29869: Allow underscores in numeric literals in lib2to3. #1109 bpo-29869: Allow underscores in numeric literals in lib2to3. #1119 [3.6] bpo-29869: Allow underscores in numeric literals in lib2to3. (GH-1119) #1122 bpo-29869: Add Nevada Sanchez to Misc/ACKS #1125

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-04-14.01:31:24.462>
created_at = <Date 2017-03-22.00:12:25.319>
labels = ['3.7', 'type-bug', 'expert-2to3']
title = 'Underscores in numeric literals not supported in lib2to3.'
updated_at = <Date 2017-04-14.01:31:24.461>
user = 'https://github.com/nevsan'

bugs.python.org fields:

activity = <Date 2017-04-14.01:31:24.461>
actor = 'Mariatta'
assignee = 'none'
closed = True
closed_date = <Date 2017-04-14.01:31:24.462>
closer = 'Mariatta'
components = ['2to3 (2.x to 3.x conversion tool)']
creation = <Date 2017-03-22.00:12:25.319>
creator = 'nevsan'
dependencies = []
files = []
hgrepos = []
issue_num = 29869
keywords = []
message_count = 14.0
messages = ['289951', '289982', '289988', '290001', '290007', '290008', '290023', '291598', '291600', '291618', '291623', '291634', '291635', '291636']
nosy_count = 4.0
nosy_names = ['georg.brandl', 'serhiy.storchaka', 'Mariatta', 'nevsan']
pr_nums = ['752', '1109', '1119', '1122', '1125']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29869'
versions = ['Python 3.6', 'Python 3.7']

[nevsan]

The following should work in Python 3.6

from lib2to3.pgen2 import driver
from lib2to3 import pytree
from lib2to3 import pygram

_GRAMMAR_FOR_PY3 = pygram.python_grammar_no_print_statement.copy()
parser_driver = driver.Driver(_GRAMMAR_FOR_PY3, convert=pytree.convert)
tree = parser_driver.parse_string('100_1\n', debug=False)

[serhiy-storchaka]

Python uses more strong rules for underscores in numerical literals. Underscores are acceptable between digits and between the base prefix and digit. For example the regular expression for hexadecimals is r'0[xX]?[\da-fA-F]+(?:[\da-fA-F]+)*[lL]?'.

Underscores also are acceptable in the exponent of float literals:

Exponent = r'[eE][-+]?\d+(?:_\d+)*'

[nevsan]

The existing regular expressions weren't actually strict enough as is, so I made them even more correct. In particular, we must have at least one digit following 0[xXbBoO] and must be before any underscores.

I have a small set of test cases to examine correctness of these regular expressions: https://gist.github.com/nevsan/7fc78dc61d309842406d67d6839b9861

[birkenfeld]

In particular, we must have at least one digit following 0[xXbBoO] and must be before any underscores.

This is not true (but your test file does the right thing).

[nevsan]

Thanks, it seems I misspoke. Glad I tested it!

[serhiy-storchaka]

I suggest to use my regular expression for haxadedecimals. Your regular expression starves from catastrophic backtracking. Compare two examples:

re.match(r'0[xX]?[\da-fA-F]+(?:[\da-fA-F]+)*[lL]?'+r'\b', '0x'+'0'*100+'z')
re.match(r'0xX+[lL]?'+r'\b', '0x'+'0'*100+'z')

[nevsan]

Good point. Updated.

[Mariatta]

New changeset 97a40b7 by Mariatta (Nevada Sanchez) in branch '3.6':
bpo-29869: Allow underscores in numeric literals in lib2to3. (GH-752)
97a40b7

[Mariatta]

New changeset 84c2d75 by Mariatta in branch '3.6':
Revert "bpo-29869: Allow underscores in numeric literals in lib2to3. (GH-752)" (GH-1109)
84c2d75

[Mariatta]

Nevada Sanchez, please create your PR against the master branch.
Once that is merged, we will backport it to the 3.6 branch.

Thanks :)

[Mariatta]

New changeset a6e395d by Mariatta (Nevada Sanchez) in branch 'master':
bpo-29869: Allow underscores in numeric literals in lib2to3. (GH-1119)
a6e395d

[Mariatta]

New changeset 2cdf087 by Mariatta in branch '3.6':
[3.6] bpo-29869: Allow underscores in numeric literals in lib2to3. (GH-1119) (GH-1122)
2cdf087