Incompatibility with Python 3.7+

[fmatter]

Don't know about the details, but the example setup with bibtex.py does not work with Python 3.8:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/citeproc/source/bibtex/latex/__init__.py", line 98, in dispatch
    yield handle_macro(tokens, macros)
  File "/usr/local/lib/python3.8/site-packages/citeproc/source/bibtex/latex/__init__.py", line 124, in handle_macro
    name = parse_macro_name(tokens)
  File "/usr/local/lib/python3.8/site-packages/citeproc/source/bibtex/latex/__init__.py", line 137, in parse_macro_name
    while tokens.peek().type == CHARACTER and tokens.peek().value.isalpha():
  File "/usr/local/lib/python3.8/site-packages/citeproc/source/bibtex/latex/__init__.py", line 72, in peek
    self._next_token = next(self._tokens)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "bibtex.py", line 19, in <module>
    bib_source = BibTeX('xampl.bib')
  File "/usr/local/lib/python3.8/site-packages/citeproc/source/bibtex/bibtex.py", line 80, in __init__
    self.add(self.create_reference(key, entry))
  File "/usr/local/lib/python3.8/site-packages/citeproc/source/bibtex/bibtex.py", line 236, in create_reference
    csl_fields = self._bibtex_to_csl(bibtex_entry)
  File "/usr/local/lib/python3.8/site-packages/citeproc/source/bibtex/bibtex.py", line 106, in _bibtex_to_csl
    value = self._parse_string(value)
  File "/usr/local/lib/python3.8/site-packages/citeproc/source/bibtex/bibtex.py", line 202, in _parse_string
    output += make_string(string)
  File "/usr/local/lib/python3.8/site-packages/citeproc/source/bibtex/bibtex.py", line 190, in make_string
    unlatexed = parse_latex(string, self.preamble_macros)
  File "/usr/local/lib/python3.8/site-packages/citeproc/source/bibtex/latex/__init__.py", line 19, in parse_latex
    for result in dispatch(tokens, macros):
RuntimeError: generator raised StopIteration

I was able to fix it by catching StopIterations in __init__.py in the latex module.

[yarikoptic]

From brief look at the code I think the issue still persists. Probably a unittest is due to be added as part of the fixing since no CI fails observed

[brechtm]

I was also having a look at this one. In trying to fix it, I'm running into other issues, e.g. parsing top-level TeX macros. Apparently, ancient changes in my working directory indicate that I already ran into this. Not sure why this example didn't fail before. Fixing the top-level macro issue will require a deep dive into the TeX parsing code...

[mcepl]

@fmatter Are you able to write your issue into some kind of reproducer script or a test case, please?

I have tried to package citeproc-py for openSUSE and its test-suite passes without any problems on all interpreters up to 3.12.