Nested multi-line expression will lead to "compile()" fails

[xxm]

BPO	42918
Nosy	@gvanrossum, @ambv, @lysnikolaou, @hongweipeng, @pablogsal, @miss-islington
PRs	bpo-42918: Improve build-in function compile() in mode 'single' #29934 [3.10] bpo-42918: Improve build-in function compile() in mode 'single' (GH-29934) #30040

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-12-27.16:17:23.694>
created_at = <Date 2021-01-13.06:01:57.716>
labels = ['interpreter-core', 'build', '3.10', '3.11']
title = 'Nested multi-line expression will lead to "compile()" fails'
updated_at = <Date 2021-12-27.16:17:23.690>
user = 'https://bugs.python.org/xxm'

bugs.python.org fields:

activity = <Date 2021-12-27.16:17:23.690>
actor = 'lukasz.langa'
assignee = 'none'
closed = True
closed_date = <Date 2021-12-27.16:17:23.694>
closer = 'lukasz.langa'
components = ['Interpreter Core']
creation = <Date 2021-01-13.06:01:57.716>
creator = 'xxm'
dependencies = []
files = []
hgrepos = []
issue_num = 42918
keywords = ['patch']
message_count = 9.0
messages = ['384995', '384999', '385000', '385001', '385003', '386542', '408269', '409231', '409232']
nosy_count = 7.0
nosy_names = ['gvanrossum', 'lukasz.langa', 'lys.nikolaou', 'hongweipeng', 'pablogsal', 'miss-islington', 'xxm']
pr_nums = ['29934', '30040']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue42918'
versions = ['Python 3.10', 'Python 3.11']

[xxm]

For build-in function compile() in mode 'single', only single statement can be well compiled. If we compile multiple statements, the parser will raise a syntaxError. Seeing the following two programs, In program 1, 2 statement are compiled. So the parser raises a Syntax error. It's the expected output. However, if we insert a nested multi-line assignment in this code(see program 2), 3 statements are compiled. We expect the parser also raise a Sytax error. But it' not.

Program 1 ( with expected results)
===================================

code1 = """
a = 1
b = 2
"""
c = compile(code1, "", "single")

===================================

Traceback (most recent call last):
  File "/home/xxm/Desktop/nameChanging/report/test1.py", line 641, in <module>
    c = compile(code1, "", "single")
  File "", line 2
    a = 1
         ^
SyntaxError: multiple statements found while compiling a single statement

Program 2 (with unexpected results)
=================================

code2 = """
c ='''
d=1
'''
a = 1
b = 2
"""
c = compile(code2, "", "single")

=================================

Expected out for program 2: Raise a syntaxError too， But it's not.

> python -V
Python 3.10.0a2
>uname -a
Linux xxm-System-Product-Name 4.15.0-64-generic #73~16.04.1-Ubuntu SMP Fri Sep 13 09:56:18 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

[gvanrossum]

What does the bytecode for c in program 2 look like?

Also, how do you find all these bugs? Are you developing a new fuzzier?

[xxm]

The bytecode of c is as following:

          0 LOAD_CONST               0 (1)
          2 STORE_NAME               0 (d)
          4 LOAD_CONST               1 (None)
          6 RETURN_VALUE

===============================================

Yes, my team is working on developing a new fuzzier for the CPython.

Also, we are trying to apply it on the newest version of CPython.

We will carefully analyze all bugs before reporting them on the tracker.

Hope our work does not bother you too much.

[gvanrossum]

It’s good work! Is it open source?

[xxm]

For sure! As soon as we validate this technique, we will open source it on GitHub.

[lysnikolaou]

Thanks a lot for the report, Xinmeng! This is a bug in how we're checking for bad single statements in bad_single_statement and it's not trivial to fix due to the string shenanigans we're doing there in newline_in_string, which does not account for many edge cases. I'll try to dig a bit deeper during the weekend.

[ambv]

New changeset 28179aa by Weipeng Hong in branch 'main':
bpo-42918: Improve build-in function compile() in mode 'single' (GH-29934)
28179aa

[ambv]

New changeset 576e38f by Miss Islington (bot) in branch '3.10':
bpo-42918: Improve built-in function compile() in mode 'single' (GH-29934) (GH-30040)
576e38f

[ambv]

Thanks! ✨ 🍰 ✨