ast.literal_eval does not accept strings with leading whitespaces

[gousaiyang]

BPO	41887
Nosy	@gvanrossum, @terryjreedy, @pablogsal, @miss-islington, @isidentical, @gousaiyang
PRs	bpo-41887: omit leading whitespaces on ast.literal_eval #22469 [3.9] bpo-41887: omit leading spaces/tabs on ast.literal_eval (GH-22469) #22533

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-10-04.00:56:32.242>
created_at = <Date 2020-09-30.01:46:57.482>
labels = ['3.8', 'type-feature', 'library', '3.9', '3.10']
title = 'ast.literal_eval does not accept strings with leading whitespaces'
updated_at = <Date 2020-10-04.00:56:32.241>
user = 'https://github.com/gousaiyang'

bugs.python.org fields:

activity = <Date 2020-10-04.00:56:32.241>
actor = 'gvanrossum'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-04.00:56:32.242>
closer = 'gvanrossum'
components = ['Library (Lib)']
creation = <Date 2020-09-30.01:46:57.482>
creator = 'gousaiyang'
dependencies = []
files = []
hgrepos = []
issue_num = 41887
keywords = ['patch']
message_count = 8.0
messages = ['377687', '377711', '377722', '377843', '377853', '377882', '377909', '377911']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'terry.reedy', 'pablogsal', 'miss-islington', 'BTaskaya', 'gousaiyang']
pr_nums = ['22469', '22533']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue41887'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[gousaiyang]

ast.literal_eval does not accept code with leading whitespaces, while eval accepts them, which is an inconsistency.

>>> import ast
>>> eval(' 1')
1
>>> ast.literal_eval(' 1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/ast.py", line 62, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/local/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 1
    1
IndentationError: unexpected indent

[isidentical]

Looks like skipping the leading whitespace is an undocumented behavior for eval() introduced back in 1992 (f08ab0a). I don't think that bringing it to the literal_eval is a good idea, at least after this much of a time. If so, both of them should be explicitly documented or at least mentioned in the source code (to avoid confusion).

[gvanrossum]

Hm, I'm not sure. ast.literal_eval() does accept trailing whitespace, and embedded whitespace.

>>> ast.literal_eval("- ( 1\n)\n")
-1
>>> 

So I think it should start accepting leading whitespace too (but only in a feature release, so 3.10).

[terryjreedy]

The doc for literal_eval says "evaluate ... a string containing a Python literal or container display." To me, ' 1' qualifies, just as it does as an expression for eval().

The exception comes from parsing raising IndentationError with leading whitespace even when the mode is 'eval' rather than 'exec'. This surprised me. Eval() gets strips the beginning of the string before it is parsed. If parsing remains as is, I agree that doing the same for literal_eval strings. But why should not parsing remove indents for 'eval' mode?