Fix tests which exit before returning a figure or use unittest.TestCase
#171
Conversation
Rather than overriding the low level `pytest_pyfunc_call`, we can wrap the test function in a wrapper that stores its return value to the plugin object.
ConorMacBride
force-pushed
the
fix-unittest
branch
from
38fb7cc
to
32b9a12
Compare
If the return value hasn't been added to the dictionary, it means that the user's test didn't reach the end. This means that it raised an exception or was otherwise stopped early by pytest (skipped, failed, etc). In any case, we should not proceed with figure testing and use the result that pytest has already determined. Fixes matplotlib#172
|
I've added tests in 32b9a12 which reproduce the bug (and various similar bugs) reported in #172. You can see that these tests fail as expected: https://github.com/matplotlib/pytest-mpl/actions/runs/2583532536 I've added a fix for this bug in d84892b, and you can see that the new tests now pass: https://github.com/matplotlib/pytest-mpl/actions/runs/2583592229 In this PR the user's test function is now wrapped in a wrapper that adds the figure to a dictionary with a key which is the test name. If for whatever reason the user's test function exits early, nothing will be added to the dictionary. Therefore, if a key with the current test name doesn't exist in the dictionary, we skip image comparison testing and let pytest handle the error/skip etc. as usual. |
ConorMacBride
changed the title
unittest.TestCaseunittest.TestCase
|
Got 2nd approval from @Cadair in a message. |
Documents kwargs for plotting functions and adds sensible kwargs to those that were lacking them (such as figsize) - Document kwargs, according to [matplotlib-style documentation](https://stackoverflow.com/questions/62511086/how-to-document-kwargs-according-to-numpy-style-docstring) - add kwargs to plotting functions missing sensible/useful kwargs - add check for unused kwargs to most plotting functions - add plot comparisons to artifact uploaded to github - bump `pytest-mpl` version to fix [bug](matplotlib/pytest-mpl#171) where if `unittest.TestCase` is used, it did not actually run those tests correctly resolves #178 , #244
Fixes #170
Currently pytest-mpl relies on the
pytest_pyfunc_callhook to intercept the figure for testing from the test function. However, when usingunittest.TestCasethis hook is not used. Rather than adding a specific fix forunittest.TestCase, in this PR I have refactored pytest-mpl such that, instead of using thepytest_pyfunc_callhook, within thepytest_runtest_callhook pytest-mpl will,Noneinstead, (pytest will requireNoneto be returned in a future version.)yieldto pytest so pytest can call the test function (either with itspytest_pyfunc_call, or otherwise in the case ofunittest.TestCase),self),I've added tests to ensure pytest-mpl can both pass and fail as expected when testing using
unittest.TestCase.Just to be sure pytest-mpl now works with a variety of different testing setups, before we make a bugfix release I want to add tests to
tests/subtestsfor testing classes. I'll do that in a separate PR.