Extension error with python 3.7 and Callable type hint

[caenrigen]

Hi! First of all thank you for the scanpydoc.elegant_typehints looks very nice, we are adopting it for a new project.

I am running into an issue that seems to happen only for python 3.7.6+ when trying to use the Callable type hint. I have also verified that with python 3.8+ things are building ok.

For context my extension list in conf.py:

extensions = [
    "sphinx.ext.autodoc",  # auto document docstrings
    "sphinx.ext.napoleon",  # autodoc understands numpy docstrings
    # load after napoleon, improved compatibility with type hints annotations
    "sphinx_autodoc_typehints",
    "sphinx.ext.viewcode",
    "sphinx.ext.intersphinx",
    "sphinx.ext.autosectionlabel",
    "sphinx-jsonschema",
    "sphinx_rtd_theme",
    "sphinx.ext.mathjax",
    "nbsphinx",
    "jupyter_sphinx",
    "sphinxcontrib.blockdiag",
    "sphinx_togglebutton",
    "scanpydoc.elegant_typehints"
]

Problematic code:

from __future__ import annotations

import typing

def test_function(call: typing.Callable):
    """
    Some doc

    Parameters
    ----------
    call:
        some docs
    """

Running sphinx mode will raise an exception and not build the docs:

[autodoc] getattr(_, 'test_function')
[autodoc] => <function test_function at 0x134459680>
[app] emitting event: 'autodoc-before-process-signature'(<function test_function at 0x134459680>, False)
[app] emitting event: 'autodoc-process-signature'('function', 'quantify.analysis.fitting_models.test_function', <function test_function at 0x13445968
[app] emitting event: 'autodoc-process-docstring'('function', 'quantify.analysis.fitting_models.test_function', <function test_function at 0x13445968
[app] emitting event: 'build-finished'(ExtensionError("Handler <function process_docstring at 0x10a621cb0> for event 'autodoc-process-docs

Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/events.py", line 110, in emit
    results.append(listener.handler(self.app, *args))
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx_autodoc_typehints.py", line 385, in process_docstring
    annotation, fully_qualified=app.config.typehints_fully_qualified)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/scanpydoc/elegant_typehints/formatting.py", line 103, in format_annotation
    annot_fmt = format_both(annotation, fully_qualified)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/scanpydoc/elegant_typehints/formatting.py", line 119, in format_both
    f":annotation-terse:`{_escape(_format_terse(annotation, fully_qualified))}`\\ "
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/scanpydoc/elegant_typehints/formatting.py", line 48, in _format_terse
    args = get_args(annotation)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/typing_extensions.py", line 2070, in get_args
    if get_origin(tp) is collections.abc.Callable and res[0] is not Ellipsis:
IndexError: tuple index out of range

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/application.py", line 352, in build
    self.builder.build_update()
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 299, in build_update
    len(to_build))
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 311, in build
    updated_docnames = set(self.read())
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 418, in read
    self._read_serial(docnames)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 439, in _read_serial
    self.read_doc(docname)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/builders/__init__.py", line 479, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/io.py", line 223, in read_doc
    pub.publish()
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/core.py", line 218, in publish
    self.settings)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/io.py", line 128, in read
    self.parse()
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/parsers.py", line 102, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 171, in run
    input_source=document['source'])
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/statemachine.py", line 242, in run
    context, state, transitions)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2769, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 395, in new_subsection
    node=section_node, match_titles=True)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 282, in nested_parse
    node=node, match_titles=match_titles)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/statemachine.py", line 242, in run
    context, state, transitions)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2769, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 395, in new_subsection
    node=section_node, match_titles=True)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 282, in nested_parse
    node=node, match_titles=match_titles)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/statemachine.py", line 242, in run
    context, state, transitions)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2342, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
    return method(self, expmatch)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2097, in directive
    directive_class, match, type_name, option_presets)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
    result = directive_instance.run()
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/ext/autodoc/directive.py", line 146, in run
    documenter.generate(more_content=self.content)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/ext/autodoc/__init__.py", line 902, in generate
    self.document_members(all_members)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/ext/autodoc/__init__.py", line 785, in document_members
    check_module=members_check_module and not isattr)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/ext/autodoc/__init__.py", line 899, in generate
    self.add_content(more_content)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/ext/autodoc/__init__.py", line 575, in add_content
    for i, line in enumerate(self.process_doc(docstrings)):
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/ext/autodoc/__init__.py", line 537, in process_doc
    self.options, docstringlines)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/application.py", line 454, in emit
    return self.events.emit(event, *args, allowed_exceptions=allowed_exceptions)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/events.py", line 118, in emit
    (listener.handler, name), exc) from exc
sphinx.errors.ExtensionError: Handler <function process_docstring at 0x10a621cb0> for event 'autodoc-process-docstring' threw an exception (exception: tuple index out of range)

Extension error:
Handler <function process_docstring at 0x10a621cb0> for event 'autodoc-process-docstring' threw an exception (exception: tuple index out of range)

Any idea what is causing this? Is there a way to fix this?

I am aware of sphinx_automodapi.smart_resolver (and that there is some recent progress in sphinx e.g., sphinx-doc/sphinx#9026) which would solve the main reason we adopted scanpydoc.elegant_typehints but it also looks better so it would be nice to keep using it.

[caenrigen]

This turned out to be an incorrect usage of the Callable type which apparently is not a problem on more recent versions of python.

In case someone run into similar issues, the correct usage needs to be, e.g.,:

def test_function(call: typing.Callable[[int], str]):
    """
    Some doc

    Parameters
    ----------
    call:
        some docs
    """

Tip: when running sphinx to build the docs you can use the -vv at the end of the command to output in a more verbose mode: sphinx-build -M html "." "_build" -vv. This might help sometimes debugging the docs build.

Cheers

PS thank you for the beautiful extension

[flying-sheep]

Oh! Sorry I missed this! Good that it resolved itself.

[caenrigen]

Oh! Sorry I missed this! Good that it resolved itself.

no problem, thank you for the reply :)

As a side note, it would be great if the extension was able to pinpoint the location of Exceptions in some more user-friendly way, and maybe with some less cryptic error. I think this has been the biggest struggle in my team when people try to build our docs while making use of this extension. They often forget (or have not yet learned it) to specify the type hints fully (e.g. Dict instead of Dict[Any, Any]) which seems to break the docs build completely without giving much information about the real issue.

Perhaps a try-except block for type hints processing would be useful to convert such cases into sphinx warnings to prevent the docs build failure.

In any case, I understand maintaining this repo is already quite some work, thank you for the efforts ✌️

[flying-sheep]

In this case it seems to be a typing_extensions bug. The docs of get_args say:

For unsupported objects return […] () correspondingly.

Therefore typing_extensions violates that contract by throwing an exception instead of returning ().

I think I catch all cases where anything kinda valid is specified, but I don’t want to catch real bugs like that one. If you find an exception when something like this is specified and no bug is to blame, then I’ll make that a warning!

---

I also think reading Python stack traces should be a skill anyone working with the language should learn, but maybe I’m missing something. For illustration, this is how I read the above stack trace:

First I start at the bottom:

  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/events.py", line 118, in emit
    (listener.handler, name), exc) from exc
sphinx.errors.ExtensionError: Handler <function process_docstring at 0x10a621cb0> for event 'autodoc-process-docstring' threw an exception (exception: tuple index out of range)

Extension error:
Handler <function process_docstring at 0x10a621cb0> for event 'autodoc-process-docstring' threw an exception (exception: tuple index out of range)

Ah, so there’s an error when processing a autodoc directive. tuple index out of range is not helpful, let’s scroll up …

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/sphinx/cmd/build.py", line 280, in build_main
    app.build(args.force_all, filenames)

I see, so just a bit more and hopefully we see the real stacktrace:

  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/scanpydoc/elegant_typehints/formatting.py", line 48, in _format_terse
    args = get_args(annotation)
  File "/usr/local/anaconda3/envs/dclab/lib/python3.7/site-packages/typing_extensions.py", line 2070, in get_args
    if get_origin(tp) is collections.abc.Callable and res[0] is not Ellipsis:
IndexError: tuple index out of range

Ah so elegant_typehints expects get_args to return () here as the python docs say, but typing_extensions seems to have a bug where it assumes that res is not of length 0 at that point.

This is a bug in typing_extensions that should be filed.

[caenrigen]

(for actionable solution about the bug skip to the end of this comment)

In this case it seems to be a typing_extensions bug. The docs of get_args say:

For unsupported objects return […] () correspondingly.

Therefore typing_extensions violates that contract by throwing an exception instead of returning ().

This is insightful, thank you for the guidance!

I think I catch all cases where anything kinda valid is specified, but I don’t want to catch real bugs like that one. If you find an exception when something like this is specified and no bug is to blame, then I’ll make that a warning!

Good point, I agree that bugs should not be silenced.

---

I also think reading Python stack traces should be a skill anyone working with the language should learn, but maybe I’m missing something.

I certainly agree with you. The two things that I did not illustrate well, so my bad there:

The first is that when usually building the docs no one uses the -vv option, and took me while to discover it. This creates the problem that building the docs yields only the following (in the case of this bug):

$ make html
Running Sphinx v3.3.1
loading pickled environment... failed
failed: unsupported pickle protocol: 5
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
loading intersphinx inventory from https://qcodes.github.io/Qcodes/objects.inv...
loading intersphinx inventory from https://xarray.pydata.org/en/stable/objects.inv...
loading intersphinx inventory from https://numpy.org/doc/stable/objects.inv...
loading intersphinx inventory from https://matplotlib.org/stable/objects.inv...
loading intersphinx inventory from https://lmfit.github.io/lmfit-py/objects.inv...
loading intersphinx inventory from https://dateutil.readthedocs.io/en/stable/objects.inv...
loading intersphinx inventory from https://quantify-quantify-core.readthedocs-hosted.com/en/develop/objects.inv...
loading intersphinx inventory from https://docs.scipy.org/doc/scipy/reference/objects.inv...
loading intersphinx inventory from https://docs.zhinst.com/zhinst-toolkit/en/latest/objects.inv...
loading intersphinx inventory from https://docs.zhinst.com/zhinst-qcodes/en/latest/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 15 source files that are out of date
updating environment: [new config] 15 added, 0 changed, 0 removed

Extension error:
Handler <function process_docstring at 0x109aa74d0> for event 'autodoc-process-docstring' threw an exception (exception: tuple index out of range)
make: *** [html] Error 2

With this output it is impossible to have a minimal clue of what could have gone wrong.
It is not my intention to blame elegant_typehints for this, of course, I am somewhat new to sphinx in general, so not sure what is the core problem here.

The second, but related difficulty is that even with the complete traceback from above there is no context regarding which particular func/class/etc. (that the documentation is being built for) is at the origin of the build failure.

For comparison if i have some broken reference, it gives context about the origin of the issue:

scheduler/helpers/waveforms.py:docstring of quantify.scheduler.helpers.waveforms.get_waveform_by_pulseid:4: WARNING: py:func reference target not found: quantify.scheduler.helpers.schedule.get_waveform

To sum up, I think that Exceptions like this should be raised and it would be much easier to understand the problem with extra context about what exactly was being processed when the issue appeared (in this case what type hint in which function signature).

---

Ah so elegant_typehints expects get_args to return () here as the python docs say, but typing_extensions seems to have a bug where it assumes that res is not of length 0 at that point.

This is a bug in typing_extensions that should be filed.

with your help in pointing the problem, I just checked and doing a

pip install --upgrade typing_extensions==3.10.0.0 # previous 3.7.4.3 still has the bug

solved the issue 🤩.

Not sure where this requirement should be filed, if it is the right place, could you specify it for elegant_typehints? (though it is strictly only a python 3.7 requirement).

[flying-sheep]

Released!

[caenrigen]

Awesome, thanks!