regex version pin breaks Python 3 support

[wichert]

Currently regex has a hard version pin on a very old version of the regex package. This version does not support Python 3, which results in flanker also not being usable in Python 3 projects.

The version pin has a comment that indicates that this is done for performance reasons. I am wondering a few things:

• Is there a benchmark script that one can run to test this performance problem? That would make it possible to discuss this with the regex developers.
• Which regex version showed this degradation?
• For many sites slow performance for flanker is not problematic. Can you consider doing the same pin/unpinned trick you added for Impossible to install due to version pins #20 for regex as well?

[cool-RR]

I am affected by this bug as well.

[russjones]

Hey guys, we are looking into this. I'll try and figure out what the issue was with the regex library as soon as possible.

[wichert]

@russjones can you give us a quick status update?

[russjones]

Hi @wichert @cool-RR, I'll discuss this with the original developer who set that requirement.

[russjones]

Hi @wichert @cool-RR, I discussed this with the original developer and the regex team, and we unpinned the version on master. It shouldn't be an issue anymore, but if you do see serious a serious performance degradation, you can always revert back to the stable branch.

[russjones]

By the way, we just updated the Benchmarks for Flanker comparing the pinned version on the stable branch and the unpinned version on master there is still a 6x performance improvement if you use the pinned version for large files.

You can test it yourself using the following script:

import sys 
import time

from flanker import mime
from flanker.mime import MimeError

def parse(data, count):
    times = []

    for _ in xrange(count):
        try:
            start = time.time()
            msg = mime.from_string(data)
            for p in msg.walk():
                if p.is_attachment():
                    p.body
            times.append(time.time() - start)
        except MimeError:
            print "Bad MIME"
            mime.recover(data)

    return times

if __name__ == "__main__":
    with open(sys.argv[1]) as f:
        data = f.read()

        times = parse(data, 50)
        print 'Mean:', reduce(lambda x, y: x + y, times) / len(times)