check_figures_equal regression from 3.2.0 to 3.2.1

[andrewzwicky]

Bug report

Bug summary

Previously, tests using check_figures_equal worked as expected on V3.2.0. When upgrading to V3.2.1, with no other changes to system, tests now fail with the error message:

file c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\matplotlib\testing\decorators.py, line 393
          @pytest.mark.parametrize("ext", extensions)
          def wrapper(*args, **kwargs):
E       fixture 'test_figure' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, get_preparsed_fingerprint_game, get_preparsed_timeline_games, get_test_events_folder, get_test_events_folder_in_progress, get_test_json_games_folder, get_test_replay_pickle_folder, get_test_unparsed_folder, get_unparsed_test_games, monkeypatch, no_cover, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, worker_id
>       use 'pytest --fixtures [testpath]' for help on them.

Code for reproduction

@check_figures_equal(extensions=["png"])
def test_mpl_321(test_figure, reference_figure):

    reference_figure.set_size_inches(8, 8)
    ref_ax = reference_figure.subplots()
    ref_ax.set_title("Test Title")
    ref_ax.pie(
        [1, 2, 3, 4, 4],
        labels=["A", "B", "C", "D", "E"],
    )


    test_figure.set_size_inches(8, 8)
    test_ax = test_figure.subplots()
    test_ax.set_title("Test Title")
    test_ax.pie(
        [1, 2, 3, 4, 4],
        labels=["A", "B", "C", "D", "E"],
    )

Actual outcome

This is the console output showing:

1. The test failing with V3.2.1
2. Uninstall V3.2.1 and installing V3.2.0
3. The test passing.

PS C:\Users\Andrew\Workspace\TripleAgent> pytest -k test_mpl_321
============================================================================================================================ test session starts =============================================================================================================================
platform win32 -- Python 3.7.5, pytest-4.5.0, py-1.8.0, pluggy-0.9.0
rootdir: C:\Users\Andrew\Workspace\TripleAgent, inifile: setup.cfg
plugins: xdist-1.29.0, forked-1.0.2, cov-2.7.1
collected 1223 items / 1222 deselected / 1 selected

triple_agent\tests\test_plot_types.py E                                                                                                                                                                                                                                 [100%]

=================================================================================================================================== ERRORS ===================================================================================================================================
____________________________________________________________________________________________________________________ ERROR at setup of test_mpl_321[png] _____________________________________________________________________________________________________________________
file c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\matplotlib\testing\decorators.py, line 393
          @pytest.mark.parametrize("ext", extensions)
          def wrapper(*args, **kwargs):
E       fixture 'test_figure' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, get_preparsed_fingerprint_game, get_preparsed_timeline_games, get_test_events_folder, get_test_events_folder_in_progress, get_test_json_games_folder, get_test_replay_pickle_folder, get_test_unparsed_folder, get_unparsed_test_games, monkeypatch, no_cover, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, worker_id
>       use 'pytest --fixtures [testpath]' for help on them.

c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\matplotlib\testing\decorators.py:393
============================================================================================================================== warnings summary ==============================================================================================================================
c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbformat\notebooknode.py:4
  c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbformat\notebooknode.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import Mapping

c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\jinja2\utils.py:485
  c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\jinja2\utils.py:485: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import MutableMapping

c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbconvert\exporters\exporter_locator.py:28
  c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbconvert\exporters\exporter_locator.py:28: DeprecationWarning: `nbconvert.exporters.exporter_locator` is deprecated in favor of `nbconvert.exporters.base` since nbconvert 5.0.
    DeprecationWarning)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================================================================================================== 1222 deselected, 3 warnings, 1 error in 17.47 seconds ============================================================================================================

Uninstall V3.2.1 -> Install V.3.2.0

PS C:\Users\Andrew\Workspace\TripleAgent> pip uninstall matplotlib
Found existing installation: matplotlib 3.2.1
Uninstalling matplotlib-3.2.1:
  Would remove:
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\matplotlib-3.2.1-py3.7-nspkg.pth
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\matplotlib-3.2.1.dist-info\*
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\matplotlib\*
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\mpl_toolkits\axes_grid1\*
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\mpl_toolkits\axes_grid\*
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\mpl_toolkits\axisartist\*
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\mpl_toolkits\mplot3d\*
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\mpl_toolkits\tests\*
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\pylab.py
Proceed (y/n)? y
  Successfully uninstalled matplotlib-3.2.1
PS C:\Users\Andrew\Workspace\TripleAgent> pip install matplotlib==3.2.0
Collecting matplotlib==3.2.0
  Using cached matplotlib-3.2.0-cp37-cp37m-win_amd64.whl (9.2 MB)
Requirement already satisfied: numpy>=1.11 in c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages (from matplotlib==3.2.0) (1.16.3)
Requirement already satisfied: python-dateutil>=2.1 in c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages (from matplotlib==3.2.0) (2.8.0)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages (from matplotlib==3.2.0) (1.0.1)
Requirement already satisfied: cycler>=0.10 in c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages (from matplotlib==3.2.0) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages (from matplotlib==3.2.0) (2.4.0)
Requirement already satisfied: six>=1.5 in c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages (from python-dateutil>=2.1->matplotlib==3.2.0) (1.12.0)
Requirement already satisfied: setuptools in c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages (from kiwisolver>=1.0.1->matplotlib==3.2.0) (40.8.0)
Installing collected packages: matplotlib
Successfully installed matplotlib-3.2.0

PS C:\Users\Andrew\Workspace\TripleAgent> pytest -k test_mpl_321
============================================================================================================================ test session starts =============================================================================================================================
platform win32 -- Python 3.7.5, pytest-4.5.0, py-1.8.0, pluggy-0.9.0
rootdir: C:\Users\Andrew\Workspace\TripleAgent, inifile: setup.cfg
plugins: xdist-1.29.0, forked-1.0.2, cov-2.7.1
collected 1223 items / 1222 deselected / 1 selected

triple_agent\tests\test_plot_types.py .                                                                                                                                                                                                                                 [100%]

============================================================================================================================== warnings summary ==============================================================================================================================
c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbformat\notebooknode.py:4
  c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbformat\notebooknode.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import Mapping

c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\jinja2\utils.py:485
  c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\jinja2\utils.py:485: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import MutableMapping

c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbconvert\exporters\exporter_locator.py:28
  c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbconvert\exporters\exporter_locator.py:28: DeprecationWarning: `nbconvert.exporters.exporter_locator` is deprecated in favor of `nbconvert.exporters.base` since nbconvert 5.0.
    DeprecationWarning)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================================================================================================== 1 passed, 1222 deselected, 3 warnings in 20.79 seconds ===========================================================================================================

Expected outcome

Tests that were previously working are not expected to start failing on version update? I might be missing something, but I didn't see anything in update notes about needing to refactor tests.

Matplotlib version

• Operating system: WIN10
• Matplotlib version: V3.2.1
• Matplotlib backend (print(matplotlib.get_backend())): TkAgg
• Python version: 3.7.5
• Jupyter version (if applicable):
• Other libraries:

absl-py               0.7.1
Adafruit-GPIO         1.0.3
Adafruit-PureIO       0.2.3
apipkg                1.5
appdirs               1.4.3
astor                 0.7.1
astroid               2.2.5
atomicwrites          1.3.0
attrs                 19.1.0
backcall              0.1.0
beautifulsoup4        4.7.1
black                 19.10b0
bleach                3.1.0
bs4                   0.0.1
certifi               2018.11.29
chardet               3.0.4
Click                 7.0
colorama              0.4.1
colored               1.3.93
coverage              4.5.3
cycler                0.10.0
decorator             4.3.0
defusedxml            0.5.0
entrypoints           0.3
execnet               1.6.0
future                0.17.1
gast                  0.2.2
grpcio                1.20.1
gym                   0.12.1
h5py                  2.9.0
idna                  2.8
ipykernel             5.1.0
ipython               7.2.0
ipython-genutils      0.2.0
ipywidgets            7.4.2
iso-639               0.4.5
iso3166               0.9
isodate               0.6.0
isort                 4.3.17
jedi                  0.13.2
Jinja2                2.10.1
json5                 0.9.3
jsonpickle            1.2
jsonschema            3.0.1
jupyter-client        5.2.4
jupyter-console       6.0.0
jupyter-core          4.4.0
jupyterlab            2.0.1
jupyterlab-server     1.0.7
Keras                 2.2.4
Keras-Applications    1.0.7
Keras-Preprocessing   1.0.9
kiwisolver            1.0.1
lazy-object-proxy     1.3.1
lxml                  4.3.2
Markdown              3.1
MarkupSafe            1.1.1
matplotlib            3.2.0
mccabe                0.6.1
mistune               0.8.4
mock                  2.0.0
more-itertools        7.0.0
mss                   4.0.3
nbconvert             5.4.1
nbformat              4.4.0
nose                  1.3.7
notebook              5.7.8
numpy                 1.16.3
opencv-contrib-python 4.0.0.21
opencv-python         4.1.0.25
pandas                0.25.0
pandocfilters         1.4.2
parso                 0.3.1
pathspec              0.6.0
pbr                   5.2.0
pickleshare           0.7.5
Pillow                5.4.1
pip                   20.0.2
pluggy                0.9.0
prometheus-client     0.6.0
prompt-toolkit        2.0.7
protobuf              3.7.1
py                    1.8.0
py-spy                0.1.10
PyAutoGUI             0.9.42
pycryptodome          3.7.2
PyGetWindow           0.0.4
pyglet                1.3.2
Pygments              2.3.1
pylint                2.3.1
PyMsgBox              1.0.6
pyparsing             2.4.0
pypiwin32             223
PyRect                0.1.4
pyrsistent            0.14.11
PyScreeze             0.1.20
PySocks               1.6.8
pytesseract           0.2.6
pytest                4.5.0
pytest-cov            2.7.1
pytest-forked         1.0.2
pytest-runner         4.4
pytest-xdist          1.29.0
python-dateutil       2.8.0
PyTweening            1.0.3
pytz                  2019.2
pywin32               224
pywinpty              0.5.5
PyYAML                5.1
pyzmq                 18.0.1
qtconsole             4.4.3
regex                 2019.8.19
requests              2.21.0
scipy                 1.3.0
selenium              3.141.0
Send2Trash            1.5.0
setuptools            40.8.0
simplejson            3.17.0
six                   1.12.0
snakeviz              2.0.0
soupsieve             1.8
spotipy               2.4.4
streamlink            1.3.0
tabulate              0.8.3
tensorboard           1.13.1
tensorflow            1.13.1
tensorflow-estimator  1.13.0
termcolor             1.1.0
terminado             0.8.2
testpath              0.4.2
toml                  0.10.0
tornado               6.0.4
traitlets             4.3.2
typed-ast             1.4.0
urllib3               1.24.1
vulture               1.0
wcwidth               0.1.7
webencodings          0.5.1
websocket-client      0.54.0
Werkzeug              0.15.2
wheel                 0.33.1
widgetsnbextension    3.4.2
wrapt                 1.11.1

[tacaswell]

As part of #16795 (which was backport of #15199) we made it possible to stack other fixtures onto functions decorated wiith check_figures_equal but it changed to passing the figures in as keywords (and doing a bit of signature surgery on the wrapper function to let pytest find and feed it the required other fixtures.

We could tweak that surgery a bit and go back to passing the figures as positional, but that introduces a miss-match between how all of the other pytest fixtures work. I think is is better to have special names rather than special positions.

If you change the names to fig_test and fig_ref it should work on all versions of Matplotlib and we should document that.

[andrewzwicky]

That's correct, just tried with those renames and it works! Thanks for the quick reply.