add \z as a synonym for \Z in Python REs for standardization

[mstevenbrown]

Feature or enhancement

Proposal:

Hello - I’m with the Austin Common Standards Revision Group - the joint technical working group established to develop and maintain the core open systems interfaces that are the POSIX™ 1003.1 (and former 1003.2) standards, ISO/IEC 9945, and the core of the Single UNIX Specification.

We have had a request to unify/rationalize the regex behaviors for “anchor at string beginning” (^ is the closest in POSIX) and “anchor at string end” ($ is the closest in POSIX). A description of this problem in depth can be found here and a table that scopes the varied solutions across varying languages can be found here.

Our working group has come to the conclusion that \A and \z are widely implemented across many ecosystems and are the most “standard” solution to the issue. We are asking if the Python community would consider adding “\z” as a synonym for “\Z” in their regex lexicon.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/proposal-add-z-as-a-synonym-for-z-in-python-res-for-standardization/90378/1

Linked PRs

• gh-133306: Support \z as a synonym for \Z in regular expressions #133314
• gh-133306: Use \z instead of \Z in regular expressions in the stdlib #133337
• gh-133306: Use \z instead of \Z in fnmatch.translate() and glob.translate() #133338

[david-a-wheeler]

+1, I'm very supportive!

Currently it's easy to make a mistake using regexes when checking inputs for security; see Correctly Using Regular Expressions. Many platforms interpret \Z as "optional newline followed by end-of-string" including Java, .NET/C#, Perl, PCRE, PHP (using PCRE), and Ruby. Many developers aren't aware that regex syntax varies between programming languages, and LLMs "borrow" regexes from other languages without necessarily fixing them. If everyone supported \A for beginning and \z for the end, we could at least have unified regular expression syntax for this common case (beginning of string .. end of string).

Thanks!

[collinfunk]

Hi @serhiy-storchaka, before you spend too much time implementing this I think you should be aware that there is some discussion on the glibc mailing list [1]. Because it is a proposed standardization, it is still subject to change. Paul Eggert, a glibc maintainer, wants to standardize \` and \' which is already in glibc and other implementations.

[1] https://inbox.sourceware.org/libc-alpha/d357ebe7-9aef-41f7-98da-5fa891f8064a@cs.ucla.edu/T/#m6dcdfc0466b896647fce7be5569d8d39af797125

[sethmlarson]

As the author of a blog post about this footgun, I am in complete support of this proposal! Thank you for opening this issue.

[serhiy-storchaka]

\` and \' is a terrible option.

• Since \' usually happens at the end of the regular expression, see what it looks like: r'\`"[^"]+"\''. It is difficult to see where the end of the string. Even if you use double quotes, adjacent ' and " are hard to red: r"\`'[^']+'\'".
• On GitHub and other programmer communication sites which use backquotes to mark a code, it is difficult to use for code containing a backquote.
• There is a historical ban of using backquotes in Python syntax.

Even if \` and \' be standardized, \A is already supported in Python, and \z is supported in many other engines. \A and \z will left a preferable syntax for these anchors.

[mstevenbrown]

Hi @serhiy-storchaka, before you spend too much time implementing this I think you should be aware that there is some discussion on the glibc mailing list [1]. Because it is a proposed standardization, it is still subject to change. Paul Eggert, a glibc maintainer, wants to standardize \` and \' which is already in glibc and other implementations.

[1] https://inbox.sourceware.org/libc-alpha/d357ebe7-9aef-41f7-98da-5fa891f8064a@cs.ucla.edu/T/#m6dcdfc0466b896647fce7be5569d8d39af797125

The glibc discussion was also started by our committee, as a part of the rationalization effort.

[collinfunk]

* On GitHub and other programmer communication sites which use backquotes to mark a code, it is difficult to use for code containing a backquote.

Yes, I experienced that in typing my original message. 😄

Anyways, I don't have super strong opinions. I just wanted to make sure the disagreement was known and considered. Thanks!

[serhiy-storchaka]

Thank you for opening the discussion and the issue, @mstevenbrown. Thank you for additional link, @@collinfunk

\z has been added as a synonym for \Z.

Some notes:

• AFAIK, \Z is an error inherited from PCRE 0.95 on which the first implementation of the re module was based. It was fixed in PCRE 2.0, and \z was added (https://www.pcre.org/original/changelog.txt), but it was stuck in Python for 27 years.

• Tcl is another programming language that uses \Z for end-of-string. Perhaps it has a similar history, if was not borrowed from Python.

• \Z may be deprecated and removed in the future, but not before all supported versions of Python support \z. It is less likely that \Z will be re-introduced with a Perl-compatible semantic (which is equivalent to (?=\n?\z)).

• \` and \' are very inconvenient for Python.

  • \' followed by the closing quote is hard to read. In a non-raw string literal it should be spelled as \\' or \\\', it is easy to make mistake.
  • \` can be confused with \' or a grit on a screen (especially if it was "typographically enhanced", e.g. https://docutils.sourceforge.io/docs/user/smartquotes.html).
  • It is difficult to use \` on GitHub and other Markdown-based forums.
  • It is difficult to use \' in a shell script or command line.
  • Currently, \` and \' mean literal ` and '. Many users escape punctuation characters in regular expressions if they are unsure whether escaping is necessary. This is a breaking change and would require transitional period of emitting FutureWarning.

  So, even if \` and \' are standardized and their support is added to Python (after a few painful years), \A and \z will remain the preferred ways to use these anchors in Python.