Major reworking of Python 2.5.2 re module

[mrabarnett]

BPO	3825
Nosy	@birkenfeld, @terryjreedy, @gpshead, @amauryfa, @pitrou, @giampaolo
Superseder	bpo-2636: Adding a new regex module (compatible with re)
Files	regex_2.5.2.diff test_re.py: Regexp Test Suite (from Lib/test/test_re.py) with Atomic Grouping regex_2.6rc2.diff regex_2.6rc2+1.diff regex_2.6rc2+2.diff regex_2.6rc2+3.diff regex_2.6rc2+4.diff regex_2.6rc2+5.diff regex_2.6rc2+6.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-08-21.23:46:11.931>
created_at = <Date 2008-09-09.21:07:52.768>
labels = ['expert-regex', 'type-bug']
title = 'Major reworking of Python 2.5.2 re module'
updated_at = <Date 2010-08-21.23:46:11.913>
user = 'https://bugs.python.org/mrabarnett'

bugs.python.org fields:

activity = <Date 2010-08-21.23:46:11.913>
actor = 'georg.brandl'
assignee = 'none'
closed = True
closed_date = <Date 2010-08-21.23:46:11.931>
closer = 'georg.brandl'
components = ['Regular Expressions']
creation = <Date 2008-09-09.21:07:52.768>
creator = 'mrabarnett'
dependencies = []
files = ['11484', '11499', '11532', '11543', '11553', '11554', '11559', '11585', '11587']
hgrepos = []
issue_num = 3825
keywords = ['patch']
message_count = 31.0
messages = ['72910', '72915', '72916', '72920', '72921', '72933', '72953', '72965', '73162', '73184', '73189', '73191', '73255', '73261', '73262', '73272', '73274', '73279', '73280', '73466', '73472', '73524', '73543', '73546', '73548', '73583', '73584', '73684', '73690', '73708', '114610']
nosy_count = 9.0
nosy_names = ['effbot', 'georg.brandl', 'terry.reedy', 'gregory.p.smith', 'amaury.forgeotdarc', 'pitrou', 'giampaolo.rodola', 'timehorse', 'mrabarnett']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = None
status = 'closed'
superseder = '2636'
type = 'behavior'
url = 'https://bugs.python.org/issue3825'
versions = ['Python 2.7']

[mrabarnett]

This is a major reworking of the re module in Python 2.5.2.

Added atomic groups.
Added possessive quantifiers.
Lookbehinds can now be variable length.
Typically x2 faster.

More changes to follow.

[benjaminp]

Very interesting! Have you seen bpo-3626? Another thing to note is that
this will have to wait for 2.7 before it could potentially be integrated
into the trunk.

[pitrou]

Hi,

This looks impressive. You should really work from the current SVN
trunk, not from the 2.5 sources, as there were some additions to the re
module (mainly, a bytecode verifier contributed by Google).

Also, if it can be split into several functionally independent patches,
it will certainly help reviewing and (perhaps) integrating.

Last remark: we are currently in the last phases of the release process
for 2.6 and 3.0, so your work will probably not get a lot of attention
in the next few weeks. Don't be discouraged though!

[gpshead]

This sounds really neat. but as Anotine said it'll be several weeks
before any of us can give this serious attention. Definitely update to
trunk and base your work off of that.

quick comments:

Your _sre.c diff appears to remove and replace the entire file. I don't
think thats actually what you did. Chances are that diff is a result of
changed newline f lea. Can you please make sure your file uses the
previous line ending style so that the diff is more managable.

Also, please include unit test updates to validate all new functionality
in Lib/test/test_re.py.

thanks for your efforts! -gps

[gpshead]

weird typo: s/f lea/formats/

[mrabarnett]

Corrected the diff file. I worked from Python 2.5.2 because that's what
I'm currently using. I'll work from the trunk in future.

[amauryfa]

The correct link is bpo-2636.
Is it the same work?

[mrabarnett]

This is different work from a different author than bpo-2636. I've
submitted what I've done so far in case my computer gets hit by a bus.
:-) I still have more work to do on it, so I'm not concerned that it
might not get any attention for a while.

[terryjreedy]

Atomic groups and possessive quantifiers appear to be relatively new:
http://en.wikipedia.org/wiki/Regular_expressions
for instance, has no mention of either that I found.

http://www.regular-expressions.info/atomic.html
http://www.regular-expressions.info/possessive.html
seemed pretty clear to me. If they accurately describe what you are
adding, they might be a starting point for the needed new manual
sections. They should include a warning about carefully ordering
alternatives lest one prune too much.

[pitrou]

By the way, the patch must be pretty incomplete, since there are almost
no changes to _sre.c. Am I missing something?

[mrabarnett]

Corrected the diff file, again. :-(

The atomic groups and possessive quantifiers are as described at
http://www.regular-expressions.info.

[effbot]

A bit more information on the changes to the core engine that are
responsible for the 2x speedup (on what?) would be nice to have, I think
(especially since you seem to have removed the KMP prefix scanner).

(Isn't there a RE benchmark suite somewhere under tests?)

[timehorse]

Well, I implemented this months ago, but have been busy with other
things so I haven't updated in a while. I noticed that the current
version is missing my patches for Atomic Grouping / Possessive
Qualifiers and a number of other patches I added in bpo-2636 , but I do
have working test cases and documentation updates for all the features
I've so far implemented as well as splitting my work into separate
sub-issues to make individual selection easier -- though with a number
of my modifications, I found that there are SO MANY co-dependencies
between, say, an engine modification (item 9) and adding Atomic Grouping
/ Possessive Qualifiers (item 1) and using shared Engine Constants (item
10) that I need a branch for Atomic, a branch for Atomic + Engine Mod 1,
Atomic + Engine Mod 2, Atomic + Shared Constants, Atomic + Engine Mod 1
+ Shared Constants AND Atomic + Engine Mod 2 + Shared Constants, and
those were just THREE item co-dependencies. My code is all off of the
trunk line for 2.6 and is currently available via my Bazaar repository
under https://code.launchpad.net/~timehorse, where you can access any
source tree via the bazaar version control client. The main reason I
got stumped in my development which might otherwise have implemented ALL
the issues I intended by now is that very situation I just described
where development of new features is NOT mutually independent. You can
see by all my branches that the multiplicity of A or B or C is just
nightmarish, but what had to be done to keep issues independent.

Anyway, I'm looking forward to having a look at your suggestions and
think we may take best advantage with combining our work visa vi these
things; after all, there's no point re-inventing the wheel.

Thanks again for your contribution, Matthew!