rply fails in new release of pypy

[Bystroushaak]

There were some changes in the pypy's regular expression engine. rply stopped working after commit ac140c11bea3.

Error:

[translation:info] 2.7.10 (5.1.2+dfsg-1~16.04, Jun 16 2016, 17:37:42)
[PyPy 5.1.2 with GCC 5.3.1 20160413]
[platform:msg] Set platform with 'host' cc=None, using cc='gcc', version='Unknown'
[translation:info] Translating target as defined by src/tinySelf/target
[translation] translate.py configuration:
[translation] [translate]
    targetspec = src/tinySelf/target
[translation] translation configuration:
[translation] [translation]
    gc = incminimark
    gctransformer = framework
    list_comprehension_operations = True
    withsmallfuncsets = 5
[translation:info] Annotating&simplifying...
[2c] {translation-task
starting annotate
[translation:info] with policy: rpython.annotator.policy.AnnotatorPolicy
.............[60] translation-task}

[Timer] Timings:
[Timer] annotate                       --- 3.4 s
[Timer] ========================================
[Timer] Total:                         --- 3.4 s
[translation:info] Error:
   File "/home/bystrousak/Plocha/tests/pypy/rpython/translator/goal/translate.py", line 318, in main
    drv.proceed(goals)
   File "/home/bystrousak/Plocha/tests/pypy/rpython/translator/driver.py", line 554, in proceed
    result = self._execute(goals, task_skip = self._maybe_skip())
   File "/home/bystrousak/Plocha/tests/pypy/rpython/translator/tool/taskengine.py", line 114, in _execute
    res = self._do(goal, taskcallable, *args, **kwds)
   File "/home/bystrousak/Plocha/tests/pypy/rpython/translator/driver.py", line 278, in _do
    res = func()
   File "/home/bystrousak/Plocha/tests/pypy/rpython/translator/driver.py", line 315, in task_annotate
    s = annotator.build_types(self.entry_point, self.inputtypes)
   File "/home/bystrousak/Plocha/tests/pypy/rpython/annotator/annrpython.py", line 92, in build_types
    return self.build_graph_types(flowgraph, inputs_s, complete_now=complete_now)
   File "/home/bystrousak/Plocha/tests/pypy/rpython/annotator/annrpython.py", line 140, in build_graph_types
    self.complete()
   File "/home/bystrousak/Plocha/tests/pypy/rpython/annotator/annrpython.py", line 229, in complete
    self.complete_pending_blocks()
   File "/home/bystrousak/Plocha/tests/pypy/rpython/annotator/annrpython.py", line 224, in complete_pending_blocks
    self.processblock(graph, block)
   File "/home/bystrousak/Plocha/tests/pypy/rpython/annotator/annrpython.py", line 398, in processblock
    self.flowin(graph, block)
   File "/home/bystrousak/Plocha/tests/pypy/rpython/annotator/annrpython.py", line 501, in flowin
    self.consider_op(op)
   File "/home/bystrousak/Plocha/tests/pypy/rpython/annotator/annrpython.py", line 653, in consider_op
    resultcell = op.consider(self)
   File "/home/bystrousak/Plocha/tests/pypy/rpython/flowspace/operation.py", line 104, in consider
    return spec(annotator, *self.args)
   File "/home/bystrousak/Plocha/tests/pypy/rpython/flowspace/operation.py", line 189, in specialized
    return impl(*[annotator.annotation(x) for x in other_args])
   File "/home/bystrousak/Plocha/tests/pypy/rpython/annotator/unaryop.py", line 949, in simple_call
    return self.analyser(self.s_self, *args)
   File "/home/bystrousak/Plocha/tests/pypy/rpython/tool/descriptor.py", line 18, in __call__
    return self.im_func(firstarg, *args, **kwds)
   File "/usr/lib/pypy/dist-packages/rply/lexergenerator.py", line 141, in method_matches
    model.SomeInteger(nonneg=True),
   File "/home/bystrousak/Plocha/tests/pypy/rpython/annotator/bookkeeper.py", line 572, in emulate_pbc_call
    return self.pbc_call(pbc, args, emulated=emulated)
   File "/home/bystrousak/Plocha/tests/pypy/rpython/annotator/bookkeeper.py", line 535, in pbc_call
    results.append(desc.pycall(whence, args, s_previous_result, op))
   File "/home/bystrousak/Plocha/tests/pypy/rpython/annotator/description.py", line 284, in pycall
    inputcells = self.parse_arguments(args)
   File "/home/bystrousak/Plocha/tests/pypy/rpython/annotator/description.py", line 269, in parse_arguments
    (self.name, e.getmsg()))
[translation:ERROR] AnnotatorError: 

signature mismatch: __init__() takes exactly 5 arguments (6 given)


Occurred processing the following simple_call:
      (AttributeError getting at the binding!)
    match_0 = simple_call(v0, v1, v2)

In <FunctionGraph of (rply.lexer:34)LexerStream.next at 0x5177168>:
Happened at file /usr/lib/pypy/dist-packages/rply/lexer.py line 43

==>             match = rule.matches(self.s, self.idx)
                if match:

Known variable annotations:
 v0 = SomeBuiltinMethod(analyser=<rpython.tool.descriptor.InstanceMethod object at 0x0000000005d69830>, methodname='matches', s_self=SomeRule())
 v1 = SomeChar(const='1', no_nul=True)
 v2 = SomeInteger(const=0, knowntype=int, nonneg=True, unsigned=False)

Processing block:
 block@131[rule_0...] is a <class 'rpython.flowspace.flowcontext.SpamBlock'> 
 in (rply.lexer:34)LexerStream.next 
 containing the following operations: 
       v0 = getattr(rule_0, ('matches')) 
       v1 = getattr(self_0, ('s')) 
       v2 = getattr(self_0, ('idx')) 
       match_0 = simple_call(v0, v1, v2) 
       v3 = bool(match_0) 
 --end--
[translation] start debugger...
> /home/bystrousak/Plocha/tests/pypy/rpython/annotator/description.py(269)parse_arguments()
-> (self.name, e.getmsg()))

IRC:

20:05 < RemoteFox> I have a slightly lame question - which of the branches are stable?
20:05 < cfbolz> RemoteFox: default (for CPython 2.7 compat) and py3.5
20:05 < RemoteFox> I was building my rpython project against the default, but it now can not translate
20:05 < cfbolz> what's the error?
20:07 < RemoteFox> https://gist.github.com/Bystroushaak/2913687c0bcac672bba9ca58cb3d5d18
20:07 < RemoteFox> last revision where it works is 94157
20:08 < cfbolz> RemoteFox: are you sure that's not a change in rply?
20:09 < RemoteFox> when I update the repo to the 94157, it translates
20:09 < RemoteFox> when I then swicht to the latest default, it doesn't
20:10 < RemoteFox> *switch
20:11 < RemoteFox> it may be a bug in rply, but I find strange that it would manifest itself between two revisons of pypy / rpython
20:12 < cfbolz> annoying
20:12 < cfbolz> no, I have a vague memory that we changed some stuff in the regular expression engine of rpython
20:12 < cfbolz> maybe that's the problem
20:13 -!- nunatak [~nunatak@unaffiliated/nunatak] has joined #pypy
20:13 < cfbolz> RemoteFox: can you see whether it is broken in ac140c11bea3
20:14 < cfbolz> and whether it works in b437cad15ce6
20:14 < RemoteFox> yeah, it is broken in ac140c11bea3
20:15 < RemoteFox> and yeah, works in b437cad15ce6
20:15 < cfbolz> ok, so I fear somebody needs to fix rply
20:16 < cfbolz> (it was really a severe miscompilation in the regular expression engine, which we couldn't easily fix without an API change)
20:16 < RemoteFox> oh, in which commits?
20:17 < cfbolz> ac140c11bea3
20:17 < RemoteFox> maybe I can just update my lexer's regexps
20:17 < RemoteFox> I will create a issue in rply
20:17 < RemoteFox> but I have my doubts whether it will be fixed
20:18 < cfbolz> I'll take a look
20:18 < cfbolz> why? rply is essentially dead?
20:19 < RemoteFox> I don't know, I can see some commits from this month, but there has been no reaction on my other issue
20:19 < cfbolz> Alex_Gaynor: you around?
20:19 <@Alex_Gaynor> cfbolz: yes
20:20 < cfbolz> Alex_Gaynor: I guess rply is not really on your agenda much, right?
20:20 <@Alex_Gaynor> cfbolz: uhh, I haven't spent a lot of time on it lately *reads scrollback*, it looks like rsre's API changed and I hven't fixed rply for it
20:21 < cfbolz> Alex_Gaynor: I can try to take a look
20:21 <@Alex_Gaynor> cfbolz: that'd be awesome, I'm happy to review/merge a PR if you have the time to figure out how the rsre API changed
20:22 < cfbolz> Alex_Gaynor: I did the change, so I should be able to ;-)
20:22 <@Alex_Gaynor> haha, perfect :D
20:22 < cfbolz> yeah, I see the test failures
20:22 < cfbolz> let me see 
20:49 < cfbolz> RemoteFox: I'll probably won't finish tonight, can you please file a bug and cc me? 
20:49 < cfbolz> So I don't forget 

Highlighting @cfbolz as requested.

[cfbolz]

See pull request #76

[alex]

Fixed!

[Bystroushaak]

Thanks!