FEATURE: run pytest in Jupyter notebooks

[pylang]

nose.tools offers functions for testing assertions, e.g. assert_equal(). These functions are callable in a Jupyter notebook (REPL) and produce a detailed output if an error is raised. I've looked through the pytest docs, but those relative invocation methods seem to focus on the commandline.

I request an interactive, nose-like feature in pytest, i.e. invoke pytest in a Jupyter notebook and generate the pytest, detailed output for each cell.

# Jupyter cell
def func(x):
    return x + 1

def test_answer():
    assert func(3) == 5

# Out: pytest output

======= FAILURES ========
_______ test_answer ________

    def test_answer():
>       assert func(3) == 5
E       assert 4 == 5
E        +  where 4 = func(3)

test_sample.py:5: AssertionError
======= 1 failed in 0.12 seconds ========

[RonnyPfannschmidt]

i had prior discussions about enabling this with @userzimmermann

[pylang]

For reference, there are a few plugins that run tests on full .ipynb files ,e.g. nbval, but I am unaware of a procedures that actually run pytest inside a notebook. @akaihola attempts to resolve this barrier with ipython_pytest, an extension that uses Jupyter cell magics to express the pytest console output:

# Jupyter cell
%%pytest

def test_my_stuff():
    assert 42 == 42

His implementation appears concise. However, rather than inserting multiple cell magics throughout the notebook in various test cells, it would be favorable to initiate the pytest outputs globally. I am not certain what an efficient approach would be. Perhaps a single builtin magic, similar to %matplotlib that presently initiates different plotting backends, e.g. %pytest, %pytest <mode> --> pytest output for cells that contain valid tests ...

Ideas are welcome. Many thanks.

[akaihola]

@pylang, another shortcoming in ipython_pytest compared to my fork of ipython_nose (original by @taavi) is that I couldn't find a way to run tests in-process and inject the notebook's environment into the test.

In other words, what ipython_nose allows me to do is prepare the test environment in multiple separate notebook cells. I often import required modules in one cell, create some test data in another cell, and define the actual test in yet another cell.

This approach works great in ipython_nose and the %%nose directive, since it shoves all globals from the notebook environment into the dynamically generated test module.

With ipython_pytest, I need to include all imports, test data preparation and test functions in a single cell. Also, I can't share imports and data between tests in different cells.

I tried to study pytest internals to make ipython_pytest work like ipython_nose, but gave up pretty quickly since it seemed to either be impossible or require lots and lots of hacking. Do you think it would be possible with reasonable effort?

[userzimmermann]

Hi @pylang @akaihola :)

As @RonnyPfannschmidt pointed out I am also - and still ;) - very interested in this feature!

I will also try to come up with some concept asap... Or we try to work on some concept together somehow... Sharing effort to make it reasonable ;)

[jonnor]

Is the Assertion rewriter that introspects and annotates the exceptions thrown by the tests available?
With that one can easily collect basic tests in the notebook manually and call them to get nice errors.

[nicoddemus]

AFAIK the assertion rewriter only works on .py files, it might be possible to refactor its logic to support rewriting the assertions transparently inside Jupyter notebooks

[slayoo]

Just for the record, this solution seems to work:
https://github.com/chmp/ipytest/blob/master/example/Magics.ipynb

[chmp]

I am the author of ipytest and just stumbled upon this issue.

For reference: ipytest supports assertion rewriting by using either the run_pytest magic or the rewrite_asserts magic. As in

%%run_pytest

def test_foo():
    assert [] == [1, 2, 3]

If there is interest in integration of this functionality in pytest, I would be happy to help.

[chmp]

@pylang, for testing a whole notebook you could use:

# cell1
import ipytest.magics

# cell 2
%%rewrite_asserts
... # define tests

# cell 3
%%rewrite_asserts
... # define more tests

# cell 3
%%run_pytest -qq
pass

[nicoddemus]

Hi @chmp, thanks for chipping in!

I see that you are using the rewrite_asserts function directly:

https://github.com/chmp/ipytest/blob/master/ipytest/magics.py#L43

If you managed to use that successfully, I believe that's the way to go. It also doesn't need any changes to pytest itself. 👍

[chmp]

While I have to admit, that it did seem a bit hacky to use pytest internals, it works quite nicely in practice. The hardest part was getting stack traces to work :) (I typed up the details here). I have been using pytest from inside notebooks pretty regulary the last couple of months and did not run into any problems so far.

Edit: I changed the implementation. It is now using the ast_transformers and works transparently for the user once activated. The use of magics is no longer required.

[jfthuong]

@slayoo

Just for the record, this solution seems to work:
https://github.com/chmp/ipytest/blob/master/example/Magics.ipynb

FYI: link is now a 404.

[slayoo]

Here it is: https://github.com/chmp/ipytest/blob/299623a1a94fd58ed7a5230a53bc7219c8ae2ce6/example/Magics.ipynb

[pylang]

Perhaps this should be a new issue, but I noticed the new subtests plugin. This part caught my attention:

        ...
            with self.subTest("custom message", i=i):
                self.assertEqual(i % 2, 0)

The result is pytest-flavored tracebacks by way of context managers.

---

Might it be possible to make a context manager that hooks into the traceback rewriter? Example:

with pytest.assertions:
    def test_a():
        assert 1
        assert 0

test_a()

================================== FAILURES ===================================
___________________________________ test_a ____________________________________

    def test_a():
        assert 1
>       assert 0
E       assert 0

Having a way to hook into the assertion writer, decoupled from the test runner, could more generally extend pytest-flavored tracebacks to interactive Python, i.e. the REPL, jupyter notebooks (not solely .py files).

[chmp]

Having worked a bit on this issue, my guess is a context manager will not work, as you do need to to inject the rewriter before the cell is executed. Therefore it's most likely going to be either a magic or spreading out the code between cells.

If you're happy to install ipytest as an additional dependency, you could do theses things with the current version as follows:

# cell 1
import ipytest
ipytest.config(magics=True)

# cell 2
%%rewrite_asserts

def test_a():
    assert 1
    assert 0

or

# cell 1
import ipytest
ipytest.config(rewrite_asserts=True)

# cell 2
def test_a():
    assert 1
    assert 0

# cell 3
ipytest.config(rewrite_asserts=False)

In both cases, the asserts will be pytest flavoured. You can execute test_a either in the cell it was defined in or anywhere else.

If installing ipytest does not work for you for some reason, feel free to copy out the relevant parts from it. It is not too much code and it is licensed under MIT.

As an aside: I fundamentally changed the way rewrites work. The implementation is now much more lightweight by using jupyter's ast_transformers.

[sashgorokhov]

Stumbled upon similar case. But my environment is a bit special - i am working with databricks and they have their custom "notebooks" based on ipython. So, ipytest doesnt really works for my case. What i would like to see is in general more "programmatic" approach to run tests with pytest. Like, just pass it some test function objects and let it run them. This would help to solve many similar cases where tests are done not by static files but rather generated or created in dynamic way.

[Chris-hughes10]

Hi, is this still under consideration? Being able to run pytest in jupyter out of the box would be fantastic. There seem to be some other packages which aim to do this, but it is difficult to see which ones are actively maintained!

[sashgorokhov]

@Chris-hughes10 FYI https://github.com/sashgorokhov/jupyter-pytest-2

[Chris-hughes10]

@sashgorokhov Looks quite neat! Is there any interest to try and integrate it into pytest?

[sashgorokhov]

@Chris-hughes10 pytest is not the place for hacky-whacky kind of code like mine 🌝

[RonnyPfannschmidt]

its a neat hack to give some base ideas

it would be much more neat if note books could have fixture cells, test cells, and a runtest magic,

the idea being that inside one notebook you can run the tests, but youd also be able to collect and run them with pytest

that however requires far more work to be attainble

[Chris-hughes10]

its a neat hack to give some base ideas

it would be much more neat if note books could have fixture cells, test cells, and a runtest magic,

the idea being that inside one notebook you can run the tests, but youd also be able to collect and run them with pytest

that however requires far more work to be attainble

Is this something that you would be interested in adding to the roadmap? Assuming that people are interested in contributing

[The-Compiler]

I don't think this belongs in the pytest core. It should be a pytest plugin, a Jupyter plugin, or perhaps a mixture of the two.

Another related project FWIW: testbook

[RonnyPfannschmidt]

certainly a mixture - this should be a own plugin, for both jupyter/ipython and pytest

enablement for those is still something that might make sense for a roadmap, but i currently cannot commit to help it
im happy to join the conversations, but someone else has to push it forward

[RonnyPfannschmidt]

@The-Compiler thanks for the reference, testbook looks like a nice tool to have for testing example content

[chmp]

Just for reference. For ipytest (still maintained), I moved away from using a module collector plugin and am now inject a fake module into sys.modules. As far as I can tell, everything you can do with pytest, you can do with notebooks this way (plugins, doctests, async testing, ...). Using a module collector plugin (as jupyter-pytest-2 seems to be doing) did not play nice with stuff like doctests.

In general, I think most of the functionality to run pytest inside notebooks should be doable with an external package. However, some of the details like parameterize are quite hard-wired to pytest being run from the command line (See here)

it would be much more neat if note books could have fixture cells, test cells, and a runtest magic

Using cells tags for this is afaik pretty much impossible as the kernel does not have access to this information. If you would rely on magics I think all of this functionality is doable. Simply package the impl into a function and then go the usual route of calling pytest.

[iwanb]

To add to the list, I also made a similar plugin: https://pytest-exploratory.readthedocs.io/en/latest/

The goal is to control a pytest session interactively against an existing codebase and do manual testing, rather than writing tests inside a Jupyter notebook, even if it should be possible to support both. It's implemented by executing the pytest hooks through IPython magics. It works but it's quite hacky, and as @chmp mentions there's a need for APIs to support those use-cases.

[RonnyPfannschmidt]

closing this as not planned for pytest core - we should open extra issues to create/teste enabling apis