Does xdoctest work inside a notebook?

[Sitwon]

Is there a straight-forward way to run xdoctest inside of a notebook?

I am testing with Google Colaboratory. I tried a few different way of invoking xdoctest.doctest_module() but they all threw errors.

The behavior I was expecting would be similar to the built-in doctest (first code block in the example.)

https://colab.research.google.com/drive/1oOQWUDdFxHKWNiUUjMrJXwPcMaumLy0O?usp=sharing

[Erotemic]

Hmm, I don't think there is a supported API for notebooks at this time. The reason is that the functions are not defined in a file or in a proper module (a notebook is just a big global namespace), and the current API assumes that you are testing something in one of these. Essentially, there is no registry (that I know of) that lists all of the functions in your notebook, or provides the source code to them.

However, I don't see any reason why we can't add support for that (I'm not a big user of notebooks myself, so I never considered it as a usecase). Here is a small proof of concept I wrote that allows you to execute all the doctests in a function:

def doctest_callable(func):
    """
    Example:
        >>> def inception():
        >>>     '''
        >>>     Example:
        >>>         >>> print("I heard you liked doctests")
        >>>     '''
        >>> func = inception
        >>> doctest_callable(func)
    """
    from xdoctest.core import parse_docstr_examples
    doctests = list(parse_docstr_examples(
        func.__doc__, callname=func.__name__))
    # TODO: can this be hooked up into runner to get nice summaries?
    for doctest in doctests:
        doctest.run(verbose=3)

Now if you have a notebook that defines a function with a doctest:

def myfunc():
    """
    Example:
        >>> print("TEST ME")
    """

You should be able to simply call

doctest_callable(myfunc)

I like the idea of a future version of xdoctest having an improved version of this xdoctest.doctest_callable method.

[Erotemic]

Note the latest version of xdoctest 0.14.0 now has a basic version of doctest_callable that should work right now, but ideally will be improved in the future.

[Sitwon]

It seems that this method does not work for the test case I had previously proposed. Is there a mistake in my example?
https://colab.research.google.com/drive/1oOQWUDdFxHKWNiUUjMrJXwPcMaumLy0O#scrollTo=4lGqHomSm74G&line=5&uniqifier=1

====== <exec> ======
* DOCTEST : <modpath?>::random_number:0, line 3 <- wrt source file
DOCTEST SOURCE
1 >>> type(random_number())
  <class 'int'>
3 >>> random_number() in range(1,7)
  True
DOCTEST STDOUT/STDERR
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-6-6a102d1c22af> in <module>()
     12 
     13 import xdoctest
---> 14 xdoctest.doctest_callable(random_number)

2 frames
/usr/local/lib/python3.6/dist-packages/xdoctest/doctest_example.py in run(self, verbose, on_error)
    557                             if part.compile_mode == 'eval':
    558                                 # print('test_globals = {}'.format(sorted(test_globals.keys())))
--> 559                                 got_eval = eval(code, test_globals)
    560                                 if EVAL_MIGHT_RETURN_COROUTINE:
    561                                     import types

<doctest:<modpath?>::random_number:0> in <module>()

NameError: name 'random_number' is not defined

[Erotemic]

Thanks for the report, I guess I should have tried this in a notebook itself (rather than just an IPython terminal). There seems to be an issue obtaining the global variables from a notebook. There is likely a way to detect and handle this case. I'll re-open the issue until it is fixed.

Do you know of any way to add Jupyter notebooks to an automated pytest suite? (I'm sure I can google this, and I will if I need to, but any recommendations would be nice). Once I have a fix, I'll want to make sure its tested in the unit tests.

[Sitwon]

Sorry, I'm actually relatively inexperienced with using notebooks. Notebooks just happen to be a good medium for teaching aspiring data scientists. I'd like to teach them how to document and test their code the way software engineers would, which is why I'm interested in this use case.

[Erotemic]

This is a good use case. I would love it if more data scientists actually tested their code. I'll definitely support this.

I'm working on fixing the bug with doctest_callable in #83, and based on your example, it probably makes sense to have doctest_module work as well. A goal of this project is that if it works with the builtin doctest, then it should work with xdoctest as well.

I've looked into the structure of Jupyter notebooks, and they seem like they are just running everything in a dynamic __main__ module namespace. The function xdoctest.doctest_module() already has logic to introspect which module called it, and I can add special casing for if that module is __main__, in which case I can force dynamic analysis and extract the docstrings in much the same way that the builtin doctest module does.

Do you have a timeline on this? Is this for classes in the fall? Or is this more for consulting-style teaching?

[Sitwon]

We plan to begin teaching the class to another group of students in mid September. We would be teaching the lesson on documentation and testing in early November. If it's not ready in time, it won't be the end of the world.

[Erotemic]

I'll give it a 95% probability that I'll have it done by then, and a 90% chance I'll do it this weekend.

[Erotemic]

I've got an initial version of this working in #84, there is still some code cleanup that I'd like to do, and I'd also like to have the ability to run xdoctest on a jupyter notebook from the command line. Tests are currently failing mainly because this feature isn't implemented yet.

To support this feature I've had to enable allowing live-modules to be passed to the "parse_doctestable" and related functions, which somewhat breaks the documentation (currently I'm passing the module via the modpath_or_name variable), so I'd like to update that so it is consistent.

When you are teaching this class, I'd appreciate if you could encourage the students to contribute any issues / bugfixes / documentation improvements to xdoctest.

[Erotemic]

I have this mostly finished, there is a small bug left to fix on windows.

[Erotemic]

@Sitwon I believe I have a working version of this in #85. Tomorrow I will merge that into master and then into the release branch, which will push xdoctest 0.15.0 onto pypi.

If you are able to, could you test out the branch works for your use case?

pip install git+https://github.com/Erotemic/xdoctest.git@dev/0.15.0

[Sitwon]

@Erotemic It's working for my tests now. Thank you!

[Erotemic]

@Sitwon I'm glad its working for you! The latest version has now been pushed to pypi.

Again, when you are teaching this class, I'd appreciate if you could encourage the students to contribute any issues / bugfixes / documentation improvements to xdoctest. Also, I'd appreciate if you would mention while using doctests in Jupyter notebooks may be supported, they are much more powerful when they are used as part of a proper python module (see caveats section bellow).

Here is a draft of some new docs that I will likely add to the readthedocs page.

Running Doctests in Jupyter Notebooks

You can run doctests within a Jupyter notebook in two ways:

Method 1 - Inside the notebook

Either insert this cell into your notebook:

if __name__ == '__main__':
    import xdoctest
    xdoctest.doctest_module()

This will execute any doctests for callables that are in the top-level namespace of the notebook. While you don't have to include the if __name__ block, it is better practice because it will prevent issues if you also wish to use "Method 2".

Method 2 - Outside the notebook

An alternative way to run would be using the xdoctest command line tool and pointing to the notebook file.

xdoctest path/to/notebook.ipynb

This will execute every cell in the notebook and then execute the doctest of any defined callable with a doctest.

Caveats

WARNING: in both of the above methods, when you execute doctests it will include any function / class that was defined in the notebook, but also any external library callable with a doctest that you import directly! Therefore it is best to (1) never use from <module> import * statements (in general using import * is bad practice) and (2) prefer using functions via their module name rather than importing directory. For example instead of from numpy import array; x = array([1]) use import numpy as np; x = np.array([1]).

Lastly, it is important to note that Jupyter notebooks are great for prototyping and exploration, but in practice storing algorithm and utilities in Jupyter notebooks is not sustainable (for some of these reasons). Reusable code should eventually be refactored into a proper pip-installable Python package where the top level directory contains a setup.py and a folder with a name corresponding to the module name and containing an __init__.py file and any other package python files. However, if you write you original Jupyter code with doctests, then when you port your code to a proper package the automated tests come with it! (And the above warning does not apply to statically parsed python packages).