Python 3.4 Recursion Depth Error

[notconfusing]

I am using mediawiki utilities xml dump parser (python3 only). https://pypi.python.org/pypi/mediawiki-utilities/0.2.1

When I try to parse any pages, am getting

  File "/usr/local/lib/python3.4/dist-packages/mwparserfromhell/string_mixin.py", line 112, in __getattr__
    return getattr(self.__unicode__(), attr)
  File "/usr/local/lib/python3.4/dist-packages/mwparserfromhell/nodes/text.py", line 38, in __unicode__
    return self.value
  File "/usr/local/lib/python3.4/dist-packages/mwparserfromhell/nodes/text.py", line 49, in value
    return self._value
RuntimeError: maximum recursion depth exceeded while calling a Python object

[earwig]

Can you give a specific page and mwparser function call that cause the error? I can't reproduce it.

[notconfusing]

I will try to reproduce it. Unfortunately the error is happening within a
multiprocessing map-reduce job, that does not fully re-raise errors. So I
must fix that before being able to report further.

Max Klein
http://notconfusing.com/

On Fri, May 9, 2014 at 10:04 AM, Ben Kurtovic notifications@github.comwrote:

Can you give a specific page and mwparser function call that cause the
error? I can't reproduce it.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/71#issuecomment-42689401
.

[earwig]

@notconfusing any news on this one, or should I close it?

[notconfusing]

@earwig I haven't produced anything new on this, so if you think it's "aged" out, then feel free to close it.

BTW, have you experimented with running mwparserfromhell on "stackless python" which purports not have these issues?

[earwig]

@notconfusing No, I haven't tried Stackless Python, but my understanding was that it's solving a different issue – microthreading, rather than having an infinitely long call stack? Either way, it doesn't seem like a proper solution (since the parser shouldn't be recursing that deeply in the first place...) but I am a little curious. Will look into it more.

[borzunov]

I've helped @nkruglikov today who stubled upon a similar problem. He used multiprocessing.imap_unordered to iterate through texts, parse them with mwparserfromhell.parse in parallel, and do other transformations. He got the following traceback:

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 429, in _handle_results
    task = get()
  File "/usr/lib/python3.5/multiprocessing/connection.py", line 253, in recv
    return ForkingPickler.loads(buf.getbuffer())
  File "/usr/local/lib/python3.5/dist-packages/mwparserfromhell/string_mixin.py", line 111, in __getattr__
    return getattr(self.__unicode__(), attr)
  File "/usr/local/lib/python3.5/dist-packages/mwparserfromhell/nodes/text.py", line 38, in __unicode__
    return self.value
  File "/usr/local/lib/python3.5/dist-packages/mwparserfromhell/nodes/text.py", line 49, in value
    return self._value

  [...]
  
  File "/usr/local/lib/python3.5/dist-packages/mwparserfromhell/string_mixin.py", line 111, in __getattr__
    return getattr(self.__unicode__(), attr)
  File "/usr/local/lib/python3.5/dist-packages/mwparserfromhell/nodes/text.py", line 38, in __unicode__
    return self.value
  File "/usr/local/lib/python3.5/dist-packages/mwparserfromhell/nodes/text.py", line 49, in value
    return self._value
RecursionError: maximum recursion depth exceeded

Eventually, it turned out that there were incorrect texts, and parse raised ValueError on them. This exception contained some information about source texts (in particular, StringMixin subclasses).

Since processes don't share common address space, Python uses pickle to send information between them. This includes sending of occured exceptions, but StringMixins inside the ValueErrors failed to be deserialized with pickle due to a bug with infinite calls in __getattr__. This led to the very unclear traceback above.

One way to fix this problem in the library can be to make subclasses of StringMixin pickle-serializable. If this isn't possible, I think this post would be useful to people who stumbled upon the same problem anyway. The possible solution for them is to handle exceptions from mwparserfromhell inside the child process and return some serializable value indicating the error.

[earwig]

Thanks for the bug report. I think I've fixed it in 6ffdfa5.