Tests failing with Matplotlib nightly build

[ConorMacBride]

I've been trying to fix the failing tests run using the Matplotlib nightly release. We are seeing the following error in GitHub Actions:

RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`). Consider using `matplotlib.pyplot.close()`.

However, this warning might be due to figures not being closed due to the following underlying error I am seeing locally on the same tests (usually):

...../.tox/py313-mpldev-pytestdev/lib/python3.13/site-packages/matplotlib/backends/backend_mixed.py:4: in <module>
    from .backend_agg import RendererAgg
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

...[removed]...

>   from matplotlib.backends._backend_agg import RendererAgg as _RendererAgg
E   ImportError: generic_type: type "RendererAgg" is already registered!

This looks to be related to pybind11 and the fact that we are testing using pytester, which is running matplotlib multiple times in the same process. I think running pytest with --runpytest subprocess (default is inprocess) would fix this issue, but that makes the tests painfully slow.

This issue only occurs on the nightly dev wheel. I'm not sure if the issue is related to how the dev wheels are built or if something has changed in the matplotlib codebase.

[ConorMacBride]

This issue is related to pybind11 v3.0.0, which Matplotlib nightly wheels are built with and so will future releases. pybind11 has switched to multi-phase module initialization (PEP 489) which is not compatible with how pytest saves/restores sys.modules before/after pytester runs. See pybind/pybind11#5780

This should be a (temporary) fix: b66e2a9