Make plastex work with Python 3.7

[kodyvajjha]

Issue

PlasTeX is currently incompatible with python 3.7 due to changes in exception handling inside generators.
Here is the traceback:

plasTeX version 1.0
ERROR: An error occurred while building the document object in <tokens> on
   line 1 (generator raised StopIteration)
ERROR: Error while reading argument "options" of documentclass in <string>
   on line 1 (generator raised StopIteration)
ERROR: Error while expanding "documentclass" in <string> on line 1
   (generator raised StopIteration)
ERROR: An error occurred while building the document object in <string> on
   line 1 (generator raised StopIteration)
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/plasTeX/Tokenizer.py", line 386, in __iter__
    token = next(charIter)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/plastex", line 142, in <module>
    main(sys.argv)
  File "/usr/bin/plastex", line 90, in main
    tex.parse()
  File "/usr/lib/python3.7/site-packages/plasTeX/TeX.py", line 435, in parse
    for item in tokens:
  File "/usr/lib/python3.7/site-packages/plasTeX/TeX.py", line 46, in __next__
    return self._next()
  File "/usr/lib/python3.7/site-packages/plasTeX/TeX.py", line 343, in __iter__
    tokens = obj.invoke(self)
  File "/usr/lib/python3.7/site-packages/plasTeX/Base/LaTeX/Packages.py", line 36, in invoke
    a = self.parse(tex)
  File "/usr/lib/python3.7/site-packages/plasTeX/__init__.py", line 472, in parse
    **arg.options)
  File "/usr/lib/python3.7/site-packages/plasTeX/TeX.py", line 757, in readArgumentAndSource
    toks, source = self.readGrouping(spec, expanded, parentNode=parentNode)
  File "/usr/lib/python3.7/site-packages/plasTeX/TeX.py", line 909, in readGrouping
    toks = self.expandTokens(toks, parentNode=parentNode)
  File "/usr/lib/python3.7/site-packages/plasTeX/TeX.py", line 406, in expandTokens
    out = tex.parse(frag)
  File "/usr/lib/python3.7/site-packages/plasTeX/TeX.py", line 435, in parse
    for item in tokens:
  File "/usr/lib/python3.7/site-packages/plasTeX/TeX.py", line 46, in __next__
    return self._next()
  File "/usr/lib/python3.7/site-packages/plasTeX/TeX.py", line 322, in __iter__
    token = next()
  File "/usr/lib/python3.7/site-packages/plasTeX/TeX.py", line 266, in itertokens
    t = next(inputs[-1][-1])
RuntimeError: generator raised StopIteration

Description of Fix

I managed to bypass this issue by following the fix described in the link above:
A hypothetical parser

def parser(f):
    while True:
        data = next(f)
        while True:
            line = next(f)
            if line == "- end -": break
            data += line
        yield data

would need to be rewritten as

def parser(f):
    while True:
        try:
            data = next(f)
            while True:
                line = next(f)
                if line == "- end -": break
                data += line
            yield data
        except StopIteration:
            return

Actual Fix

This issue is bypassed once I apply the fix described above to the while loops in the following:

1. The function __iter__ in the file TeX.py.
2. The functions iterchars and __iter__ in the file Tokenizer.py.

I am not sure if I should raise this as a PR because this seems to be a hackish workaround.

[tiarno]

Thanks for the report. I'm not sure returning on StopIteration is all we need to do. Currently when we hit StopIteration, the end_input method is called which gets the most recent input source from the stack. I'm going to refer this to @kesmit13 so he can have a look.

[austrin]

Is there any chance this will be fixed in the near future, or should one try to work around it?

FWIW, it seems to me that the fixes in gerby-project#58 might be fine -- the calls to endInput() only seem to happen in two places in TeX.py and those are not touched.

And regardless of whether they are sufficient, they are clearly harmless -- they just make sure that StopIteration gets propagated upwards instead of being converted to a RuntimeError (so in Python <= 3.6 they should effectively be nops, and in Python >= 3.7 they should be undoing the change in behaviour).

And given that they are harmless and at least resolve some issues, perhaps it would make sense to merge those changes for now?

[pehrsoderman]

Any update on this issue?

[PatrickMassot]

Fixed in #118.