Codeop: Show warnings once during _maybe_compile

[csabella]

BPO	40807
Nosy	@terryjreedy, @vstinner, @serhiy-storchaka, @Carreau, @csabella, @miss-islington
PRs	bpo-40807: Show warnings once from codeop._maybe_compile #20486 [3.9] bpo-40807: Show warnings once from codeop._maybe_compile (GH-20486) #20639 [3.8] bpo-40807: Show warnings once from codeop._maybe_compile (GH-20486) #20640 [3.7] bpo-40807: Show warnings once from codeop._maybe_compile (GH-20486) #20641 [3.8] bpo-40807: Backport test_codeop change [GH-19670] #20646 [3.7] bpo-40807: Backport test_codeop change [GH-19670] #20647 [3.8] bpo-40807: Show warnings once from codeop._maybe_compile (GH-20486) #20672 [3.7] bpo-40807: Show warnings once from codeop._maybe_compile (GH-20486) #20673 bpo-41520: codeop no longer ignores SyntaxWarning #21838

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-06-09.13:12:26.101>
created_at = <Date 2020-05-28.13:55:08.174>
labels = ['type-bug', '3.8', '3.9', '3.10', '3.7', 'library']
title = 'Codeop: Show warnings once during _maybe_compile'
updated_at = <Date 2020-08-12.17:54:48.490>
user = 'https://github.com/csabella'

bugs.python.org fields:

activity = <Date 2020-08-12.17:54:48.490>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-06-09.13:12:26.101>
closer = 'terry.reedy'
components = ['Library (Lib)']
creation = <Date 2020-05-28.13:55:08.174>
creator = 'cheryl.sabella'
dependencies = []
files = []
hgrepos = []
issue_num = 40807
keywords = ['patch']
message_count = 11.0
messages = ['370208', '370722', '370724', '370738', '370739', '370748', '370750', '370751', '370814', '371102', '375155']
nosy_count = 6.0
nosy_names = ['terry.reedy', 'vstinner', 'serhiy.storchaka', 'mbussonn', 'cheryl.sabella', 'miss-islington']
pr_nums = ['20486', '20639', '20640', '20641', '20646', '20647', '20672', '20673', '21838']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40807'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9', 'Python 3.10']

[csabella]

When calling codeop._maybe_compile, compile is run three times. If the code being compiled causes a warning message, the warning is generated each time that compile is called, thus (possibly) showing the message three times.

See msg370163 and that issue for context.

[terryjreedy]

The purpose of code and codeop is to imitate interactive python in python. Calling compile() thrice is an implementation detail. Exposing that is a bug. On bpo-37824, Serhiy reported 3 'IDLE' bugs for 3.7 to 3.9: tripled DeprecationWarning (now joined by SyntaxWarning), printing to console (if available) instead of Shell, and an exit exception. This issue is continuation of the triplicate issue, given that the bug is not in IDLE. As a bug, I think it should still be backported.

I looked more carefully at the behavior of the standard REPL with different statement inputs. Before executing, if the statement is complete, it may raise a compile exception, usually SyntaxError, or emit one or more a warnings, once for each instance. For multiline statements, DeprecationWarnings are emitted immediately after the line with the deprecated code while SyntaxWarnings are emitted, if there is no syntax error, just before execution. SyntaxWarnings for a given slice of code are not repeated. However, at least some of these details have to be enforced in the UI code that repeatedly calls compile or _maybe_compile as lines are added, not in compile, and should not be in _maybe_compile. Both have no memory or previous calls. (I don't think that IDLE necessarily *must* do the same either.)

Looking at compile itself, it appears that warnings are not emitted if there is an exception. But I am puzzled at this in 3.9 and master.

>>> if '\e' is 1:
<stdin>:1: DeprecationWarning: invalid escape sequence \e
...
versus
>>> compile('if '\e' is 1:\n', '', 'single')
  File "<stdin>", line 1
SyntaxError: unexpected character after line continuation character

Something must cause "'\e'" to be a warning instead of an error.

This is enough to experiment with your patch applied.

[terryjreedy]

I am not sure if 3 compiles are always needed. At one time, compile required a final '\n' to not raise, but that is no longer true. And as near as I can tell, 'code without final newline\n' and 'code without final newline\n\n' always compile the same. Certainly, Shell does not need to compile without a final newline. But we can look at this in a separate issue. If IDLE's requirement is different from code.II's, I believe it could pass its own compile function.

[terryjreedy]

New changeset 052d3fc by Cheryl Sabella in branch 'master':
bpo-40807: Show warnings once from codeop._maybe_compile (bpo-20486)
052d3fc

[miss-islington]

New changeset 3dfe549 by Miss Islington (bot) in branch '3.9':
bpo-40807: Show warnings once from codeop._maybe_compile (GH-20486)
3dfe549

[terryjreedy]

New changeset a5d6aba by Terry Jan Reedy in branch '3.8':
[3.8] bpo-40807: Backport test_codeop change [GH-19670]
a5d6aba

[terryjreedy]

New changeset 12d3061 by Miss Islington (bot) in branch '3.7':
[3.8] bpo-40807: Backport test_codeop change [GH-19670]
12d3061

[terryjreedy]

This is a spinoff of bpo-37824.

My above comments about compile are wrong. The example nests '' within ''. And, with freshly compiled master, I found an example where Deprecation warning, but not SyntaxWarning, accompanies SyntaxError, and which requires trailing \n.

>>> compile("if'\e' is 1: 1", '', 'single')
DeprecationWarning...
SyntaxError: unexpected EOF while parsing
>>> compile("if'\e' is 1: 1\n", '', 'single')
DeprecationWarning...
SyntaxWarning...
<code object ...>

Interactively,
>>> if'\e' is 1: 1
<stdin>:1: DeprecationWarning: invalid escape sequence \e
... 
<stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="?
>>>

In IDLE, DeprecationWarning is repeated, but that is another issue.

The secondary prompts seems like a bug, The statement is complete and entering anything but a comment is a SyntaxError. (A comment results in another ... prompt.)

[miss-islington]

New changeset c067183 by Miss Islington (bot) in branch '3.8':
bpo-40807: Show warnings once from codeop._maybe_compile (GH-20486)
c067183

[terryjreedy]

The 3.7 backport is needed but required a revised example. The changeset is 4b378ac.

PRs 20486, 20639, 20672, and 20673 are the main patch. PRs 20646 and 20647 for 3.8 and 3.7 backport a test_codeop import change made elsewhere and not backported. The alternative would have been changing our backport to the older naming convention.

[Carreau]

I think that might have introduce https://bugs.python.org/issue41520 where now warnings.simplefilter('error', SyntaxWarning) is silently ignored...