Deprecate the use of flags not at the start of regular expression

[serhiy-storchaka]

BPO	22493
Nosy	@pitrou, @ezio-melotti, @serhiy-storchaka, @timgraham
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	re_deprecate_nonstart_flags.patch re_deprecate_nested_flags.patch re_deprecate_nonstart_flags2.patch better-warning.diff better-warning-2.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-09-16.22:33:54.254>
created_at = <Date 2014-09-25.10:29:00.775>
labels = ['expert-regex', 'type-feature', 'library']
title = 'Deprecate the use of flags not at the start of regular expression'
updated_at = <Date 2018-05-27.08:53:56.829>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2018-05-27.08:53:56.829>
actor = 'aleskva'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-09-16.22:33:54.254>
closer = 'serhiy.storchaka'
components = ['Library (Lib)', 'Regular Expressions']
creation = <Date 2014-09-25.10:29:00.775>
creator = 'serhiy.storchaka'
dependencies = []
files = ['36721', '36845', '44526', '44675', '44680']
hgrepos = []
issue_num = 22493
keywords = ['patch']
message_count = 21.0
messages = ['227520', '228841', '228930', '228931', '228959', '275601', '275758', '276559', '276562', '276566', '276583', '276620', '276621', '276648', '276650', '276746', '276757', '276758', '279567', '317786', '317788']
nosy_count = 7.0
nosy_names = ['pitrou', 'ezio.melotti', 'mrabarnett', 'python-dev', 'serhiy.storchaka', 'Tim.Graham', 'aleskva']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue22493'
versions = ['Python 3.6']

[serhiy-storchaka]

The meaning of inline flags not at the start of regular expression is ambiguous. Current re implementation and regex in the V0 mode enlarge the scope to all expression. In V1 mode in regex they affect only the end of the expression.

I propose to deprecate (and then forbid in 3.7) the use of inline flags not at the start of regular expression. This will help to change the meaning of inline flags in the middle of the expression in future (in 3.8 or later).

[serhiy-storchaka]

Here is alternative, much simpler, patch. It deprecates only flags in nested subpatterns. No changes needed in tests and other stdlib modules. It is very unlikely that it is used in third party code.

[pitrou]

Here is alternative, much simpler, patch. It deprecates only flags in nested subpatterns.

That sounds a bit random. It wouldn't totally address the discrepancy with regex, would it?
MRAB, what do you think on this topic?

[mrabarnett]

I think the simplest and clearest approach from the user's point of view is just to deprecate inline flags that are not at the start of the pattern.

In practice, they almost invariably occur at the start anyway, although I do remember once seeing a pattern in which the inline flag was at the end!

[serhiy-storchaka]

That sounds a bit random. It wouldn't totally address the discrepancy with regex, would it?

No, it will not totally address the discrepancy with regex, but at least it will allow as to change the behavior of flags in subpatterns. And we always can convert a pattern to a subpattern (surround by "(?:" and ")").

For now Python re module is only one regular expression implementation in which flags in the middle of the expression affect all expression. [1]

[1] http://www.regular-expressions.info/modifiers.html

[serhiy-storchaka]

Updated patch that deprecates flags not at the start. fnmatch.translate() now uses scoped flags (bpo-433028).

[python-dev]

New changeset 31f8af1c3567 by Serhiy Storchaka in branch 'default':
Issue bpo-22493: Inline flags now should be used only at the start of the
https://hg.python.org/cpython/rev/31f8af1c3567

[timgraham]

Could we include the offending pattern in the deprecation message? I'm attaching a proposed patch. With that patch I can more easily find the offending pattern, whereas before I had no idea:

django/django/urls/resolvers.py:101: DeprecationWarning: Flags not at the start of the expression ^(?i)test/2/?$
compiled_regex = re.compile(regex, re.UNICODE)

[timgraham]

And on further investigation, I'm not sure how to fix the deprecation warnings in Django. We have a urlpattern like this:

url(r'^(?i)CaseInsensitive/(\w+)', empty_view, name="insensitive"),

The regex string r'^(?i)CaseInsensitive/(\w+)' is later substituted in this line in Django's URL resolver as the pattern:

if re.search('^%s%s' % (re.escape(_prefix), pattern), candidate_pat % candidate_subs, re.UNICODE):

It seems Django would need to extract any flags from pattern and put them at the start of the '^%s%s' string that's constructed for re.search(). I'm not sure if this can be done easily.

[mrabarnett]

@tim: Why are you using re.search with '^'? Does the pattern that's passed in ever contain '(?m)'? If not, re.match without '^' is better.