Issue recognizing my docstring

[ArneBachmannDLR]

For the following docstring I get an error:

def logTraceback(logFunction):
  r''' Logs the exception traceback to the specified log function.

  >>> try: raise Exception()  # doctest: +ELLIPSIS
  ... except Exception: logTraceback(lambda *a, **b: sys.stdout.write(a[0] + "\n", *a[1:], **b))
  Traceback (most recent call last):
  ...
  Exception
  ...
  '''
  # here is the code

Error:

running 62 test(s)
====== <exec> ======
* DOCTEST : D:\forks\HACE\autocook\autocook\base.py::logTraceback:0, line 21 <- wrt source file
DOCTEST SOURCE
1 >>> try: raise Exception()  # doctest: +ELLIPSIS
2 ... except Exception: logTraceback(lambda *a, **b: sys.stdout.write(a[0] + "\n", *a[1:], **b))
  Traceback (most recent call last):
  ...
  Exception
  ...
DOCTEST STDOUT/STDERR

Traceback (most recent call last):
  File "d:\apps\Miniforge3\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "d:\apps\Miniforge3\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "d:\apps\Miniforge3\lib\site-packages\xdoctest\__main__.py", line 172, in <module>
    retcode = main()
  File "d:\apps\Miniforge3\lib\site-packages\xdoctest\__main__.py", line 160, in main
    run_summary = xdoctest.doctest_module(modname, argv=[command], style=style,
  File "d:\apps\Miniforge3\lib\site-packages\xdoctest\runner.py", line 302, in doctest_module
    run_summary = _run_examples(enabled_examples, verbose, config,
  File "d:\apps\Miniforge3\lib\site-packages\xdoctest\runner.py", line 465, in _run_examples
    summary = example.run(verbose=verbose, on_error=on_error)
  File "d:\apps\Miniforge3\lib\site-packages\xdoctest\doctest_example.py", line 612, in run
    code = compile(
  File "<doctest:D:\forks\HACE\autocook\autocook\base.py::logTraceback:0>", line 2
    except Exception: logTraceback(lambda *a, **b: sys.stdout.write(a[0] + "\n", *a[1:], **b))
                                                                                             ^
SyntaxError: unexpected EOF while parsing

[Erotemic]

Thanks for the report! This is definitely a bug.

FWIW if you move the statements to newlines, the syntax error goes away:

    >>> try:
    ...     raise Exception()
    ... except Exception:
    ...     logTraceback(lambda *a, **b: sys.stdout.write(a[0] + "\n", *a[1:], **b))

But the example you provided is valid Python syntax, so I'll look into why it thinks there is an error.

If the body of the function isn't too complicated do you mind sharing it? I'd also like to check that the traceback is correctly matched. I don't often use the exception checker (usually I would just import pytest and do a with pytest.raises in the doctest itself to check that sort of thing), so this would be a good case to add to the test suite.

[ArneBachmannDLR]

Sure, but my compressed code style is probably not very test-case worthy:

class LogPipe(threading.Thread):
  r''' Fake file-like stream object that redirects writes to a logger instance.
  >>> stl = LogPipe(lambda *a, **b: sys.stdout.write(a[0] + "\n", *a[1:], **b))
  >>> stl.write(f"Test\n{stl.fdWrite}"); stl.close()
  Test
  4
  '''

  def __init__(_, logMethod:Callable[[str], None], ignoreEmptyLines:bool = True, encoding:str = 'utf-8'):
    ''' Setup this thread with a log method and start it. '''
    threading.Thread.__init__(_)
    _.daemon = False
    _.logMethod = logMethod
    _.ignoreEmptyLines = ignoreEmptyLines
    _.fdRead, _.fdWrite = os.pipe()
    _.pipeReader = os.fdopen(_.fdRead)
    _.encoding = encoding
    _.lines:List[str] = []  # unicode
    _.remainder = ''
    _.start()

  def fileno(_):
    ''' Return the write file descriptor of the pipe. '''
    return _.fdWrite

  def flush(_) -> None:
    if _.remainder: _.lines.append(_.remainder); _.logMethod(_.remainder); _.remainder = ''

  def write(_, what:Union[str,bytes], *args, **kwargs) -> None:
    what = what if isinstance(what, str) else cast(str, what.decode(_.encoding))
    lines = what.split('\n')
    lines[0] = _.remainder + lines[0]; _.remainder = lines[-1]  # augment first line
    lines.pop()
    for line in lines:
      if _.ignoreEmptyLines and line == '': continue
      _.lines.append(line); _.logMethod(line)

  def run(_) -> None:
    ''' Run the thread, logging everything. '''
    for line in iter(_.pipeReader.readline, ''): _.logMethod(line.strip('\n')); _.lines.append(line.strip('\n'))
    _.pipeReader.close()

  def close(_) -> None:
    ''' Close the write end of the pipe. '''
    _.flush()
    os.close(_.fdWrite)

  def _test(_):
    r''' Test the PipeLog.
    >>> logger = logging.getLogger()
    >>> stream = LogPipe(logging.debug)
    >>> stream.write(b"First\n")
    >>> stream.write(b"Second\nFourth")
    >>> stream.write(b"\nThird\na")
    >>> stream.write(b"bc")
    >>> stream.close()
    >>> print(stream.lines)
    ['First', 'Second', 'Fourth', 'Third', 'abc']
    '''
    pass


def logTraceback(logFunction):
  r''' Logs the exception traceback to the specified log function.

  >>> try: raise Exception()  # doctest: +ELLIPSIS
  ... except Exception: logTraceback(lambda *a, **b: sys.stdout.write(a[0] + "\n", *a[1:], **b))
  Traceback (most recent call last):
  ...
  Exception
  ...
  '''
  try:
    import traceback
    from .utils import LogPipe
    pipe = LogPipe(logFunction)
    traceback.print_exc(        None, pipe)  # noqa: E201
    traceback.print_stack(None, None, pipe)
    pipe.close()
  except Exception as E: error("Could not log exception traceback: '%s'" % E)

[Erotemic]

@ArneBachmannDLR I've looked into this a little bit. This is a tricky issue. It won't be a quick fix.

As a very immediate workaround you can just change the second ... to >>> , and it does work roughly as expected in that case. The reason is because the problem that is causing the error is a backwards compatibility thing with doctest itself. If you aren't using the ... it doesn't trigger it.

The fundamental issue is that this piece of code errors:

        compile('try: raise Exception\nexcept Exception: pass', mode='single', filename="")

With

  File "<string>", line 2
    except Exception: pass
                         ^
SyntaxError: unexpected EOF while parsing

Note that the following variations do not error:

        compile('try: raise Exception\nexcept Exception: pass', mode='exec', filename="")
        compile('try:\n    raise Exception\nexcept Exception:\n    pass', mode='single', filename="")
        compile('try:\n    raise Exception\nexcept Exception:\n    pass', mode='exec', filename="")

So I'm wondering if this is an actual honest-to-goodness CPython bug (this wouldn't be the first time I've run into one working on this project). It's very strange that the version of this statement with newlines and indentation works correctly, but the concise version does not in the builtin python compile function.

Normally, xdoctest just uses exec, except in the case where it is targeting that backwards compatible functionality, and in that case it uses single instead.

I could just put a workaround in that switches to exec mode whenever a statement begins with a try, and that would fix this issue, and and I'm open to falling back on that if fixing this takes too long, but I'd like to dig into this a bit more, before I add more difficult-to-maintain hacks to the already unwieldy parser (if you know of anyone with more AST experience than myself, fixing my parser would be a great exercise!).

EDIT:

It is not a CPython bug: As the docs state: https://docs.python.org/3/library/functions.html#compile "When compiling a string with multi-line code in 'single' or 'eval' mode, input must be terminated by at least one newline character. This is to facilitate detection of incomplete and complete statements in the code module"

[Erotemic]

So, maybe it wont take that long after all.

The PR with a fix is up: #107

The basic change is that any doctest part that is asked to be compiled in single mode ensures that it adds an extra newline at the end of it's source.

[ArneBachmannDLR]

Awesome!

[Erotemic]

This should be fixed in the released 0.15.9. Please let me know if there is still an issue.

[ArneBachmannDLR]

Thanks, great work! I can finally see xdoctest do a full run through all detected doctests, in beautiful color with helpful highlights!