gh-98401: Invalid escape sequences emits SyntaxWarning #99011

vstinner · 2022-11-02T16:21:32Z

A backslash-character pair that is not a valid escape sequence now generates a SyntaxWarning, instead of DeprecationWarning. For example, re.compile("\d+.\d+") now emits a SyntaxWarning ("\d" is an invalid escape sequence), use raw strings for regular expression: re.compile(r"\d+.\d+"). In a future Python version, SyntaxError will eventually be raised, instead of SyntaxWarning.

Octal escapes with value larger than 0o377 (ex: "\477"), deprecated in Python 3.11, now produce a SyntaxWarning, instead of DeprecationWarning. In a future Python version they will be eventually a SyntaxError.

codecs.escape_decode() and codecs.unicode_escape_decode() are left unchanged: they still emit DeprecationWarning.

The parser only emits SyntaxWarning for Python 3.12 (feature version), and still emits DeprecationWarning on older Python versions.
Fix SyntaxWarning by using raw strings in Tools/c-analyzer/ and wasm_build.py.

Issue: Reject invalid escape sequences (and octal escape sequences) in bytes and Unicode strings #98401

Parser/string_parser.c

vstinner · 2022-11-02T21:50:47Z

Previous attempt in 2018:

Commit: 6543912
Serhiy only changed the Python parser, not codecs: rationale
The new SyntaxWarning was seen in multiple projects: docutils (fixed), bottle, SymPy (bug in a docstring)
SyntaxWarning are only emitted once, and only when a Python file is parsed: when a cached PYC file is used, not warning is emitted
Serhiy reverted his change

vstinner · 2022-11-03T11:06:06Z

I tested the Python test suite with SyntaxWarning treated as error: it does pass.

$ PYTHONWARNINGS=error::SyntaxWarning ./python -W error::SyntaxWarning -m test -j0 -r -u all,-gui

vstinner · 2022-11-03T11:19:33Z

This issue mostly hit code defining regular expressions. Example in BeautifulSoup 3.2.2:

$ pycodestyle --select W605 $(find -name "*.py")
./BeautifulSoup.py:455:52: W605 invalid escape sequence '\d'
./BeautifulSoup.py:455:73: W605 invalid escape sequence '\w'

./BeautifulSoup.py:574:47: W605 invalid escape sequence '\d'
./BeautifulSoup.py:574:66: W605 invalid escape sequence '\w'

./BeautifulSoup.py:1080:38: W605 invalid escape sequence '\s'

./BeautifulSoup.py:1589:36: W605 invalid escape sequence '\s'

./BeautifulSoup.py:1933:16: W605 invalid escape sequence '\?'
./BeautifulSoup.py:1933:46: W605 invalid escape sequence '\?'
./BeautifulSoup.py:1935:35: W605 invalid escape sequence '\s'

Examples of code, re.compile() calls:

    BARE_AMPERSAND_OR_BRACKET = re.compile("([<>]|"
                                           + "&(?!#\d+;|#x[0-9a-fA-F]+;|\w+;)"
                                           + ")")
(...)
                      (re.compile('<!\s+([^<>]*)>'),
(...)
        xml_encoding_match = re.compile(
            '^<\?.*encoding=[\'"](.*?)[\'"].*\?>').match(xml_data)

Doc/reference/lexical_analysis.rst

serhiy-storchaka · 2022-11-03T11:25:58Z

Lib/test/test_codecs.py

        for i in range(97, 123):
            b = bytes([i])
            if b not in b'abfnrtvx':
-                with self.assertWarns(DeprecationWarning):
+                with self.assertWarns(SyntaxWarning):


SyntaxWarning is not related to codecs. It only should be emitted by the compiler.

Which warning should be emitted if SyntaxWarning is not the best choice? UnicodeWarning?

Does UnicodeWarning make sense for codecs.escape_decode()?

DeprecationWarning, as for all other deprecated features.

Parser/string_parser.c

serhiy-storchaka · 2022-11-03T11:31:54Z

I tested the Python test suite with SyntaxWarning treated as error: it does pass.

Did you remove all pyc files and regenerate frozen modules before tests? Also try to regenerate generated code using the new Python binary as PYTHON_FOR_REGEN.

vstinner · 2022-11-03T12:08:27Z

Did you remove all pyc files and regenerate frozen modules before tests? Also try to regenerate generated code using the new Python binary as PYTHON_FOR_REGEN.

I ran PYTHONWARNINGS=error::SyntaxWarning make regen-all PYTHON_FOR_REGEN=./python: it works as expected.

vstinner · 2022-11-03T12:09:10Z

@serhiy-storchaka:

Perhaps SyntaxWarning could be raised for invalid octals too. It is a more obvious error, and less common, so it may have lesser deprecation period.

Ok, I updated my PR: Use SyntaxWarning for invalid octal sequence.

A backslash-character pair that is not a valid escape sequence now generates a SyntaxWarning, instead of DeprecationWarning. For example, re.compile("\d+\.\d+") now emits a SyntaxWarning ("\d" is an invalid escape sequence), use raw strings for regular expression: re.compile(r"\d+\.\d+"). In a future Python version, SyntaxError will eventually be raised, instead of SyntaxWarning. Octal escapes with value larger than 0o377 (ex: "\477"), deprecated in Python 3.11, now produce a SyntaxWarning, instead of DeprecationWarning. In a future Python version they will be eventually a SyntaxError. codecs.escape_decode() and codecs.unicode_escape_decode() are left unchanged: they still emit DeprecationWarning. * The parser only emits SyntaxWarning for Python 3.12 (feature version), and still emits DeprecationWarning on older Python versions. * Fix SyntaxWarning by using raw strings in Tools/c-analyzer/ and wasm_build.py.

vstinner · 2022-11-03T12:39:14Z

Hum, I messed up my PR so I squashed two commits and I fixed my PR:

invalid octal sequence and invalid escape sequence now both emit SyntaxWarning (and will emit SyntaxError in the future)
codecs.escape_decode() and codecs.unicode_escape_decode() are left unchanged

I also mentioned the future convertion to SyntaxError in the doc (What's New / NEWS entries).

vstinner · 2022-11-03T15:04:02Z

@mdickinson @serhiy-storchaka @hugovk: Would you mind to review my PR?

serhiy-storchaka · 2022-11-03T15:29:34Z

Did you remove all pyc files?

find -name '*.py[co]' -exec rm -rf '{}' +

vstinner · 2022-11-03T16:12:20Z

Let me try these commands:

export PYTHONWARNINGS=error::SyntaxWarning 
git clean -fdx
./configure --with-pydebug CFLAGS=-O0
make
make regen-all PYTHON_FOR_REGEN=./python
make

./python -c 's = "\477"'

The last command fails as expected with:

  File "<string>", line 1
    s = "\477"
        ^^^^^^
SyntaxError: invalid octal escape sequence '\477'

vstinner · 2022-11-03T16:16:50Z

Oops, test_string_literals didn't work when run with ./python -W error::SyntaxWarning: it's now fixed.

I ran the test suite with:

export PYTHONWARNINGS=error::SyntaxWarning 
./python -W error::SyntaxWarning -m test -j0 -r -u all,-gui

Note: I would prefer to run the whole test suite with -Werror, but it fails even without this PR :-( Python currently emits many deprecation warnings for example :-(

Fixes this warning experienced with Python 3.12 (python/cpython#98401, python/cpython#99011): faa_cs_aan.py:690: SyntaxWarning: invalid escape sequence '\.' email_match = re.search('For Inquiries: ([0-9a-z._-]+@[0-9a-z.-]+)\.?$',

vstinner requested review from pablogsal and lysnikolaou as code owners November 2, 2022 16:21

bedevere-bot mentioned this pull request Nov 2, 2022

Reject invalid escape sequences (and octal escape sequences) in bytes and Unicode strings #98401

Closed

bedevere-bot added the awaiting core review label Nov 2, 2022

This was referenced Nov 2, 2022

gh-98401: Reject invalid escape sequences in strings #98404

Closed

Raise non-silent warning for invalid escape sequences #77093

Open

lysnikolaou reviewed Nov 2, 2022

View reviewed changes

Parser/string_parser.c Outdated Show resolved Hide resolved

serhiy-storchaka reviewed Nov 3, 2022

View reviewed changes

serhiy-storchaka approved these changes Nov 3, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting core review labels Nov 3, 2022

Fix test_string_literals

d24f799

vstinner merged commit a60ddd3 into python:main Nov 3, 2022

bedevere-bot removed the awaiting merge label Nov 3, 2022

vstinner deleted the syntax_warning branch November 3, 2022 16:53

chenhengqi mentioned this pull request Oct 31, 2023

tools/*.py: fix invalid escape sequences in regexps iovisor/bcc#4789

Merged

benoit9126 mentioned this pull request Nov 9, 2023

Syntax warning because of "\S" in Python 3.12 keygen-sh/py-machineid#9

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-98401: Invalid escape sequences emits SyntaxWarning #99011

gh-98401: Invalid escape sequences emits SyntaxWarning #99011

vstinner commented Nov 2, 2022 •

edited

vstinner commented Nov 2, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

serhiy-storchaka Nov 3, 2022

vstinner Nov 3, 2022

serhiy-storchaka Nov 3, 2022

serhiy-storchaka commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

serhiy-storchaka commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

gh-98401: Invalid escape sequences emits SyntaxWarning #99011

gh-98401: Invalid escape sequences emits SyntaxWarning #99011

Conversation

vstinner commented Nov 2, 2022 • edited

vstinner commented Nov 2, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

serhiy-storchaka Nov 3, 2022

Choose a reason for hiding this comment

vstinner Nov 3, 2022

Choose a reason for hiding this comment

serhiy-storchaka Nov 3, 2022

Choose a reason for hiding this comment

serhiy-storchaka commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

serhiy-storchaka commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 3, 2022

vstinner commented Nov 2, 2022 •

edited