New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reject invalid escape sequences (and octal escape sequences) in bytes and Unicode strings #98401
Comments
I created PR #98404 to implement this change. This change should help to catch some mistakes in regular expressions and Windows paths. Example with PR #98404 which now raises SyntaxError:
Raw strings
|
See also discussion in #77093, and the reasons that the |
When this warning was introduced, there were no any plans of making it an error in the near future. It was planned as a warning with very long period. The specific of this warning is that it is not emitted for the code containing this specific kind of bug, but only emitted for the code which does not contain a bug. But which may be in near proximity of the code which contains a bug (and does not emit a warning itself). Perhaps it is a time to make it more visible (convert it into SyntaxWarning). And then after other long period of 4-5 versions it could be converted into SyntaxError. |
It first went into Python 3.6, now EOL, so all supported Python versions have the deprecation warning. Serhiy suggested in #71551 (comment) a
So that was a suggestion of 2 releases from deprecation to syntax warning. If we do a syntax warning now, that will have already been 6 releases. I don't know what the original estimate of 3.8 -> 4.0 was? Also two releases? Guido suggested in #71551 (comment) for "several" releases before the error.
Six releases (3.6 -> 3.12) is perhaps several? The original issue also had suggestions from Victor and Guido to contact linters/PyCQA to include a warning to help projects prepare, and the original author did so. And the good news is it was added to pycodestyle (part of Flake8) as W605 in April 2018, so that's 4.5 years of linter warnings. (Thanks to this, I've fixed invalid escape sequences in several packages.) @vstinner Would it be worth running Depending on the result, it may be worth promoting to a |
Ok, let's start with replacing DeprecationWarning (silent by default) with SyntaxWarning (displayed once by default): PR #99011. |
A backslash-character pair that is not a valid escape sequence now generates a SyntaxWarning, instead of DeprecationWarning. For example, re.compile("\d+\.\d+") now emits a SyntaxWarning ("\d" is an invalid escape sequence), use raw strings for regular expression: re.compile(r"\d+\.\d+"). In a future Python version, SyntaxError will eventually be raised, instead of SyntaxWarning. Octal escapes with value larger than 0o377 (ex: "\477"), deprecated in Python 3.11, now produce a SyntaxWarning, instead of DeprecationWarning. In a future Python version they will be eventually a SyntaxError. codecs.escape_decode() and codecs.unicode_escape_decode() are left unchanged: they still emit DeprecationWarning. * The parser only emits SyntaxWarning for Python 3.12 (feature version), and still emits DeprecationWarning on older Python versions. * Fix SyntaxWarning by using raw strings in Tools/c-analyzer/ and wasm_build.py.
Fixed by a60ddd3 At the end, it remains a warning, but SyntaxWarning (showed by default) is now emitted instead of DeprecationWarning (silent by default). According to @hugovk, sadly many projects of the PyPI top 5000 contain invalid escape sequences. It will take time to update them, before considering to convert the SyntaxWarning to an SyntaxError. Thanks for everybody who helped me on making this change possible! For affected projects: just add |
In case of regular expressions, I'm wondering if we'll be forced to roll back these changes before release again due to a lot of warnings in third-party code. |
I created https://discuss.python.org/t/collaboration-on-handling-python-3-12-incompatible-changes-distutils-removal-invalid-escape-escape-etc/20721 to discuss this change. |
If you work on windows and have the habit of documenting your code inside trippelquotes, (A company requirement) you now get a warning whenever a directory name starts on a d because ex: """C:\dist\project\file.dat""" gives a SyntaxWarning. And not to mention the regexp issue that have a fix. More work to port a 3.11 project to 3.12 than moving it from python2 to python3 |
Just replace |
Is there a tool that modifies the problematic regular expression? |
Fixes this warning experienced with Python 3.12 (python/cpython#98401): faa_cs_aan.py:690: SyntaxWarning: invalid escape sequence '\.' email_match = re.search('For Inquiries: ([0-9a-z._-]+@[0-9a-z.-]+)\.?$',
Fixes this warning experienced with Python 3.12 (python/cpython#98401, python/cpython#99011): faa_cs_aan.py:690: SyntaxWarning: invalid escape sequence '\.' email_match = re.search('For Inquiries: ([0-9a-z._-]+@[0-9a-z.-]+)\.?$',
- After updating Python 3.12.0 need to add the letter r before \ the redexp python/cpython#98401. appstream-data-mobile-20240125-alt1 - Initial build in Sisyphus. - Added exclusivelist file for ALT mobile applications. - Generated appstream-data-mobile for Sisyphus.
A backslash-character pair that is not a valid escape sequence now generates a SyntaxWarning. It outputs to stderr which makes the tests to fail. See python/cpython#98401 . https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1061803
As of python/cpython#98401 (released in Python 3.12.0), invalid escape sequences emit louder warnings, as described in hallettj#89. This patch uses raw strings for compiled regexes to suppress the warnings.
As of python/cpython#98401 (released in Python 3.12.0), invalid escape sequences emit louder warnings, as described in #89. This patch uses raw strings for compiled regexes to suppress the warnings.
When I run: |
I cannot reproduce your issue. Also, this issue is closed.
|
quick update to say that the warning can be automatically fixed with ruff
I've checked some edge cases (strings with \n), it's adjusting to raw string or escaping to double slash, as needed for each string. |
In Python 3.6, invalid escape sequence were deprecated in string literals (bytes and str): issue #71551, commit 110b6fe.
What's New in Python 3.6: Deprecated Python behavior:
I propose now raises a SyntaxError, rather than a DeprecationWarning (which is silent in most cases).
Example:
Note: Python REPL ate some DeprecationWarning which makes manual testing harder. It was fixed last month by commit 426d72e in issue gh-96052.
Python 3.11 now emits a deprecation warning for invalid octal escape sequence (issue gh-81548):
Example:
The text was updated successfully, but these errors were encountered: