Invalid escape sequence DeprecationWarnings don't trigger by default

[Numerlor]

BPO	40199
Nosy	@terryjreedy, @ncoghlan, @vstinner, @serhiy-storchaka, @Numerlor, @snoopjedi
Files	gdb_deprecationwarning_session.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-04-05.20:29:24.208>
labels = ['interpreter-core', 'type-bug', '3.9', '3.10', '3.11']
title = "Invalid escape sequence DeprecationWarnings don't trigger by default"
updated_at = <Date 2021-06-03.18:25:46.751>
user = 'https://github.com/Numerlor'

bugs.python.org fields:

activity = <Date 2021-06-03.18:25:46.751>
actor = 'SnoopJeDi'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2020-04-05.20:29:24.208>
creator = 'Numerlor'
dependencies = []
files = ['50091']
hgrepos = []
issue_num = 40199
keywords = []
message_count = 9.0
messages = ['365825', '365826', '365832', '366150', '366151', '366191', '394913', '394933', '395030']
nosy_count = 7.0
nosy_names = ['terry.reedy', 'ncoghlan', 'vstinner', 'habnabit', 'serhiy.storchaka', 'Numerlor', 'SnoopJeDi']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue40199'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[Numerlor]

The current warning filter seems to filter out the compile time DeprecationWarnings that get triggered on invalid escape sequences:

import warnings

compile("'\d'", "<string>", "eval")
warnings.resetwarnings()
compile("'\d'", "<string>", "eval")

results in one
<string>:1: DeprecationWarning: invalid escape sequence \d
being printed

[serhiy-storchaka]

I get it printed two times.

Actually I get it printed four times: two time when compile the test script (use r"'\d'" or "'\\d'" to get rid of them), and other two times when call compile() inside a script.

$ ./python issue40199.py 
issue40199.py:3: DeprecationWarning: invalid escape sequence \d
  compile("'\d'", "<string>", "eval")
issue40199.py:5: DeprecationWarning: invalid escape sequence \d
  compile("'\d'", "<string>", "eval")
<string>:1: DeprecationWarning: invalid escape sequence \d
<string>:1: DeprecationWarning: invalid escape sequence \d

[Numerlor]

In what environment was that ran in?
The issue seems to exist in all the different environments that are easily available to me (win7, win10, linux under docker and on repl.it)

[terryjreedy]

On Win 10, with recently compiled 3.7.7+ and 3.9.0a5+, I get 4 warnings also.

>>> import warnings
>>> compile("'\d'", "<string>", "eval")
<stdin>:1: DeprecationWarning: invalid escape sequence \d
<string>:1: DeprecationWarning: invalid escape sequence \d
<code object <module> at 0x00A65DC0, file "<string>", line 1>
>>> warnings.resetwarnings()
>>> compile("'\d'", "<string>", "eval")
<stdin>:1: DeprecationWarning: invalid escape sequence \d
<string>:1: DeprecationWarning: invalid escape sequence \d
<code object <module> at 0x00A66190, file "<string>", line 1>

Numerior, what 3.8 binaries are you running? I suspect that this should be closed as 'out of date'.

[Numerlor]

The newest 64 bit release from python.org via the executable installer.

[terryjreedy]

OK, verified issue with installed 3.9.0a5, which must have different warning settings.

From a file, 1 warning.
<string>:1: DeprecationWarning: invalid escape sequence \d

Interactively, no warning.

>>> compile("'\d'", "<string>", "eval")
<code object <module> at 0x00000263842DFC90, file "<string>", line 1>
>>>

[habnabit]

This is definitely not windows-specific. On macos:

$ python3.9
Python 3.9.4 (default, Apr  5 2021, 01:47:16)
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> '\s'
'\\s'

[serhiy-storchaka]

Ah, there is a difference between debug and regular builds. I tested with the debug build.

[snoopjedi]

The cause of DeprecationWarning sometimes [1] not being issued is I believe because in string_parser.c [2] the module is explicitly set to NULL and the filename will be '<string>' or '<stdin>' or somesuch, which eventually that ends up being normalized to something that isn't '__main__'.

Not sure if this is stating the obvious and I don't see any general solution short of adding a filter for the special filenames, but I caught wind of this in #python on Libera, got curious, and thought I'd share what I learned here. I've also attached a gdb session showing how changing the filename affects this behavior.

Reproducing this in debug/dev contexts is definitely fraught, since the warning behavior is different.

[1] The given compile() sample, at the REPL, when using -c, or when piping input via stdin are the ones I know about
[2]

cpython/Parser/string_parser.c

Lines 19 to 20 in f3fa63e

	if (PyErr_WarnExplicitObject(PyExc_DeprecationWarning, msg, p->tok->filename,
	t->lineno, NULL, NULL) < 0) {