Misleading documentations and comments in regular expression HOWTO

[vajrasky]

BPO	18779
Nosy	@akuchling, @pitrou, @ezio-melotti, @bitdancer, @vajrasky
Files	fix_alphanumeric_and_underscore_doc_in_regex.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-02-15.02:46:23.850>
created_at = <Date 2013-08-19.07:45:16.799>
labels = ['expert-regex', 'docs']
title = 'Misleading documentations and comments in regular expression HOWTO'
updated_at = <Date 2017-02-15.02:46:23.836>
user = 'https://github.com/vajrasky'

bugs.python.org fields:

activity = <Date 2017-02-15.02:46:23.836>
actor = 'akuchling'
assignee = 'docs@python'
closed = True
closed_date = <Date 2017-02-15.02:46:23.850>
closer = 'akuchling'
components = ['Documentation', 'Regular Expressions']
creation = <Date 2013-08-19.07:45:16.799>
creator = 'vajrasky'
dependencies = []
files = ['31371']
hgrepos = []
issue_num = 18779
keywords = ['patch']
message_count = 5.0
messages = ['195611', '195617', '195618', '195627', '287809']
nosy_count = 7.0
nosy_names = ['akuchling', 'pitrou', 'ezio.melotti', 'mrabarnett', 'r.david.murray', 'docs@python', 'vajrasky']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue18779'
versions = ['Python 3.3', 'Python 3.4']

[vajrasky]

According to:

http://oald8.oxfordlearnersdictionaries.com/dictionary/alphanumeric
http://en.wikipedia.org/wiki/Alphanumeric

Alphanumeric is defined as [A-Za-z0-9]. Underscore (_) is not one of them. One of the documentation in Python (Doc/tutorial/stdlib2.rst) differentiates them very clearly:

"The format uses placeholder names formed by $ with valid Python identifiers
(alphanumeric characters and underscores). Surrounding the placeholder with
braces allows it to be followed by more alphanumeric letters with no intervening
spaces. Writing $$ creates a single escaped $::"

Yet, in documentations as well as comments in regex, we implicitely assumes underscore belongs to alphanumeric.

Explicit is better than implicit!

Attached the patch to differentiate alphanumeric and underscore in documentations and comments in regex.

This is important in case someone is confused with this code:
>>> import re
>>> re.split('\W', 'haha$hihi*huhu_hehe hoho')
['haha', 'hihi', 'huhu_hehe', 'hoho']

On the side note:
In Python code base, sometimes we write "alphanumerics" and "underscores", yet sometimes we write "alphanumeric characters" and "underscore characters". Which one again is the true way?

[pitrou]

I was wondering which doc you were alluding it, before I noticed your patch is against the regex HOWTO.
The HOWTO seems quite outdated wrt. Python 3. For example "\w" is not equivalent to "[a-zA-Z0-9_]", anymore, except with the ASCII flag.

[vajrasky]

In Lib/re.py, starting from line 77 (Python 3.4):

\w       Matches any alphanumeric character; equivalent to [a-zA-Z0-9_]
         in bytes patterns or string patterns with the ASCII flag.
         In string patterns without the ASCII flag, it will match the
         range of Unicode alphanumeric characters (letters plus digits
         plus underscore).
         With LOCALE, it will match the set [0-9_] plus characters defined
         as letters for the current locale.

The prelude is "Matches any alphanumeric character;".

Yet, in any case (bytes, string patterns with ascii flag, string patterns without the ascii flag, strings with locale), the underscore is always included.

Then why don't we change the prelude to "Matches any alphanumeric character and underscore character;"? In the description we explain the alphanumeric depending on it's unicode or not can be [A-Za-z0-9] or wider than that.

The description is already okay but the prelude is misleading readers.

[bitdancer]

The answer to the question about "alphanumerics" versus "alphanumeric characters" is that is is mostly likely context-dependent, so I'd have to see particular examples to say which I though read better. So, there is no One True Answer for this question, I think.

[akuchling]

Unfortunately making the sentences pedantically correct also makes them ungainly, and I think people generally assume that underscores are treated as a letter.