[Bug]: Figure manager is gone - AttributeError: 'NoneType' object has no attribute 'canvas'

[1kastner]

Bug summary

The figure manager is None even though everything should work nicely.

Code for reproduction

A simple example of pandas should suffice:

import pandas as pd
df = pd.DataFrame({'lab':['A', 'B', 'C'], 'val':[10, 30, 20]})
ax = df.plot.bar(x='lab', y='val', rot=0)

Actual outcome

Out of the sudden, my server CI fails with the following traceback:

 File ~/work/conflowgen/conflowgen/conflowgen/analyses/inbound_and_outbound_vehicle_capacity_analysis_report.py:77, in InboundAndOutboundVehicleCapacityAnalysisReport.get_report_as_graph(self, **kwargs)
     71 df = pd.DataFrame({
     72     "inbound volume (in TEU)": inbound_capacities,
     73     "outbound volume (in TEU)": outbound_actual_capacities,
     74     "outbound maximum capacity": outbound_maximum_capacities
     75 })
     76 df.index = [str(i).replace("_", " ") for i in df.index]
---> 77 ax = df.plot.barh()
     78 ax.set_xlabel("Capacity (in TEU)")
     79 ax.set_title("Inbound and outbound vehicle capacity analysis")

File ~/.local/lib/python3.8/site-packages/pandas/plotting/_core.py:1217, in PlotAccessor.barh(self, x, y, **kwargs)
   1133 @Appender(
   1134     """
   1135     See Also
   (...)
   1206 @Appender(_bar_or_line_doc)
   1207 def barh(self, x=None, y=None, **kwargs):
   1208     """
   1209     Make a horizontal bar plot.
   1210 
   (...)
   1215     other axis represents a measured value.
   1216     """
-> 1217     return self(kind="barh", x=x, y=y, **kwargs)

File ~/.local/lib/python3.8/site-packages/pandas/plotting/_core.py:972, in PlotAccessor.__call__(self, *args, **kwargs)
    969             label_name = label_kw or data.columns
    970             data.columns = label_name
--> 972 return plot_backend.plot(data, kind=kind, **kwargs)

File ~/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/__init__.py:71, in plot(data, kind, **kwargs)
     69         kwargs["ax"] = getattr(ax, "left_ax", ax)
     70 plot_obj = PLOT_CLASSES[kind](data, **kwargs)
---> 71 plot_obj.generate()
     72 plot_obj.draw()
     73 return plot_obj.result

File ~/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py:328, in MPLPlot.generate(self)
    326 self._args_adjust()
    327 self._compute_plot_data()
--> 328 self._setup_subplots()
    329 self._make_plot()
    330 self._add_table()

File ~/.local/lib/python3.8/site-packages/pandas/plotting/_matplotlib/core.py:386, in MPLPlot._setup_subplots(self)
    384 else:
    385     if self.ax is None:
--> 386         fig = self.plt.figure(figsize=self.figsize)
    387         axes = fig.add_subplot(111)
    388     else:

File ~/.local/lib/python3.8/site-packages/matplotlib/pyplot.py:810, in figure(num, figsize, dpi, facecolor, edgecolor, frameon, FigureClass, clear, **kwargs)
    798     _api.warn_external(
    799         f"More than {max_open_warning} figures have been opened. "
    800         f"Figures created through the pyplot interface "
   (...)
    803         f"warning, see the rcParam `figure.max_open_warning`).",
    804         RuntimeWarning)
    806 manager = new_figure_manager(
    807     num, figsize=figsize, dpi=dpi,
    808     facecolor=facecolor, edgecolor=edgecolor, frameon=frameon,
    809     FigureClass=FigureClass, **kwargs)
--> 810 fig = manager.canvas.figure
    811 if fig_label:
    812     fig.set_label(fig_label)

AttributeError: 'NoneType' object has no attribute 'canvas'
AttributeError: 'NoneType' object has no attribute 'canvas'

The first invocation is just some default invocation of pandas. Thus, I am very confident that this has nothing to do with my code. Just a few days back, the CI was working wonderfully. It is still working wonderfully locally on my computer.

Expected outcome

The figures should be displayed just like before.

Additional information

See https://github.com/1kastner/conflowgen/runs/7882122350?check_suite_focus=true for the original traceback.

It has worked before.

I have no idea why this is happening now. I was even tampering with the matplotlib backend (setting one explicitely via an env variable) but it did not have any effect on this error.

Operating system

Linux

Matplotlib Version

3.5.3

Matplotlib Backend

agg

Python version

3.10.6

Jupyter version

No response

Installation

pip

[tacaswell]

Can you reproduce this without pandas?

[jklymak]

Even with pandas I cannot reproduce this locally. But a few things did change with the backend fallbacks, so its not impossible the CI is not falling back gracefully?

[tacaswell]

new_figure_manager should never return None so something very fishy is going on.

@1kastner can you get the package lists from the last working and first broken run?

[1kastner]

My last working build on readthedocs used the following packages installed via pip:

First, the default packages of readthedocs:

Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.1 Pygments-2.13.0 alabaster-0.7.12 babel-2.10.3 certifi-2022.6.15 charset-normalizer-2.1.0 commonmark-0.8.1 docutils-0.17.1 future-0.18.2 idna-3.3 imagesize-1.4.1 mock-1.0.1 packaging-21.3 pillow-9.2.0 pyparsing-3.0.9 pytz-2022.2.1 readthedocs-sphinx-ext-2.1.8 recommonmark-0.5.0 requests-2.28.1 snowballstemmer-2.2.0 sphinx-5.1.1 sphinx-rtd-theme-1.0.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-2.0.0 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.5 urllib3-1.26.11

...and then the packages I need to build the docs:

Successfully installed SecretStorage-3.3.3 Send2Trash-1.8.0 anyio-3.6.1 apeye-1.2.0 argon2-cffi-21.3.0 argon2-cffi-bindings-21.2.0 astroid-2.11.7 asttokens-2.0.8 attrs-22.1.0 autodocsumm-0.2.9 backcall-0.2.0 beautifulsoup4-4.11.1 bleach-5.0.1 cachecontrol-0.12.11 cffi-1.15.1 commonmark-0.9.1 conflowgen-2.0.0 coverage-6.4.4 cryptography-37.0.4 cssutils-2.5.1 cycler-0.11.0 debugpy-1.6.3 decorator-5.1.1 defusedxml-0.7.1 dict2css-0.3.0 dill-0.3.5.1 domdf-python-tools-3.3.0 entrypoints-0.4 enum_tools-0.9.0.post1 et-xmlfile-1.1.0 executing-0.10.0 fastjsonschema-2.16.1 flake8-5.0.4 flake8_nb-0.5.1 fonttools-4.35.0 html5lib-1.1 importlib-metadata-4.12.0 iniconfig-1.1.1 ipykernel-6.15.1 ipython-8.4.0 ipython-genutils-0.2.0 isort-5.10.1 jedi-0.18.1 jeepney-0.8.0 jinja2-3.0.3 json5-0.9.9 jsonschema-4.10.0 jupyter-client-7.3.4 jupyter-core-4.11.1 jupyter-server-1.18.1 jupyterlab-3.4.5 jupyterlab-pygments-0.2.2 jupyterlab-server-2.15.0 kaleido-0.2.1 keyring-23.8.2 kiwisolver-1.4.4 latexcodec-2.0.1 lazy-object-proxy-1.7.1 lockfile-0.12.2 lxml-4.9.1 markdown-it-py-2.1.0 matplotlib-3.5.3 matplotlib-inline-0.1.3 mccabe-0.7.0 mdit-py-plugins-0.3.0 mdurl-0.1.2 mistune-0.8.4 msgpack-1.0.4 myst-parser-0.18.0 natsort-8.1.0 nbclassic-0.4.3 nbclient-0.6.6 nbconvert-6.5.3 nbformat-5.4.0 nbsphinx-0.8.9 nest-asyncio-1.5.5 notebook-6.4.12 notebook-shim-0.1.0 numpy-1.23.2 openpyxl-3.0.10 pandas-1.4.3 pandocfilters-1.5.0 parso-0.8.3 peewee-3.15.1 pexpect-4.8.0 pickleshare-0.7.5 pkginfo-1.8.3 platformdirs-2.5.2 plotly-5.10.0 pluggy-1.0.0 prometheus-client-0.14.1 prompt-toolkit-3.0.30 psutil-5.9.1 ptyprocess-0.7.0 pure-eval-0.2.2 py-1.11.0 pybtex-0.24.0 pybtex-docutils-1.0.2 pycodestyle-2.9.1 pycparser-2.21 pyflakes-2.5.0 pylint-2.14.5 pyrsistent-0.18.1 pytest-7.1.2 pytest-cov-3.0.0 pytest-github-actions-annotate-failures-0.1.7 python-dateutil-2.8.2 pyyaml-6.0 pyzmq-23.2.1 readme-renderer-36.0 requests-toolbelt-0.9.1 rfc3986-2.0.0 rich-12.5.1 ruamel.yaml-0.17.21 ruamel.yaml.clib-0.2.6 scipy-1.9.0 seaborn-0.11.2 setuptools-65.0.1 six-1.16.0 sniffio-1.2.0 soupsieve-2.3.2.post1 sphinx-autodoc-typehints-1.19.2 sphinx-jinja2-compat-0.1.2 sphinx-prompt-1.5.0 sphinx-tabs-3.4.0 sphinx-toolbox-3.2.0b1 sphinxcontrib-bibtex-2.4.2 stack-data-0.4.0 tabulate-0.8.10 tenacity-8.0.1 terminado-0.15.0 tinycss2-1.1.1 tomli-2.0.1 tomlkit-0.11.4 tornado-6.2 traitlets-5.3.0 twine-4.0.1 typing-extensions-4.3.0 wcwidth-0.2.5 webencodings-0.5.1 websocket-client-1.3.3 wrapt-1.14.1 zipp-3.8.1

That was one day and two hours ago. The first build afterwards happend 7 hours ago and failed with these versions:

Successfully installed Jinja2-3.1.2 MarkupSafe-2.1.1 Pygments-2.13.0 alabaster-0.7.12 babel-2.10.3 certifi-2022.6.15 charset-normalizer-2.1.0 commonmark-0.8.1 docutils-0.17.1 future-0.18.2 idna-3.3 imagesize-1.4.1 mock-1.0.1 packaging-21.3 pillow-9.2.0 pyparsing-3.0.9 pytz-2022.2.1 readthedocs-sphinx-ext-2.1.8 recommonmark-0.5.0 requests-2.28.1 snowballstemmer-2.2.0 sphinx-5.1.1 sphinx-rtd-theme-1.0.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-2.0.0 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.5 urllib3-1.26.11

...and in the second step

Successfully installed SecretStorage-3.3.3 Send2Trash-1.8.0 anyio-3.6.1 apeye-1.2.0 argon2-cffi-21.3.0 argon2-cffi-bindings-21.2.0 astroid-2.11.7 asttokens-2.0.8 attrs-22.1.0 autodocsumm-0.2.9 backcall-0.2.0 beautifulsoup4-4.11.1 bleach-5.0.1 cachecontrol-0.12.11 cffi-1.15.1 commonmark-0.9.1 conflowgen-2.0.0 coverage-6.4.4 cryptography-37.0.4 cssutils-2.5.1 cycler-0.11.0 debugpy-1.6.3 decorator-5.1.1 defusedxml-0.7.1 dict2css-0.3.0 dill-0.3.5.1 domdf-python-tools-3.3.0 entrypoints-0.4 enum_tools-0.9.0.post1 et-xmlfile-1.1.0 executing-0.10.0 fastjsonschema-2.16.1 flake8-5.0.4 flake8_nb-0.5.1 fonttools-4.35.0 html5lib-1.1 importlib-metadata-4.12.0 iniconfig-1.1.1 ipykernel-6.15.1 ipython-8.4.0 ipython-genutils-0.2.0 isort-5.10.1 jedi-0.18.1 jeepney-0.8.0 jinja2-3.0.3 json5-0.9.9 jsonschema-4.10.0 jupyter-client-7.3.4 jupyter-core-4.11.1 jupyter-server-1.18.1 jupyterlab-3.4.5 jupyterlab-pygments-0.2.2 jupyterlab-server-2.15.0 kaleido-0.2.1 keyring-23.8.2 kiwisolver-1.4.4 latexcodec-2.0.1 lazy-object-proxy-1.7.1 lockfile-0.12.2 lxml-4.9.1 markdown-it-py-2.1.0 matplotlib-3.5.3 matplotlib-inline-0.1.5 mccabe-0.7.0 mdit-py-plugins-0.3.0 mdurl-0.1.2 mistune-0.8.4 msgpack-1.0.4 myst-parser-0.18.0 natsort-8.1.0 nbclassic-0.4.3 nbclient-0.6.6 nbconvert-6.5.3 nbformat-5.4.0 nbsphinx-0.8.9 nest-asyncio-1.5.5 notebook-6.4.12 notebook-shim-0.1.0 numpy-1.23.2 openpyxl-3.0.10 pandas-1.4.3 pandocfilters-1.5.0 parso-0.8.3 peewee-3.15.1 pexpect-4.8.0 pickleshare-0.7.5 pkginfo-1.8.3 platformdirs-2.5.2 plotly-5.10.0 pluggy-1.0.0 prometheus-client-0.14.1 prompt-toolkit-3.0.30 psutil-5.9.1 ptyprocess-0.7.0 pure-eval-0.2.2 py-1.11.0 pybtex-0.24.0 pybtex-docutils-1.0.2 pycodestyle-2.9.1 pycparser-2.21 pyflakes-2.5.0 pylint-2.14.5 pyrsistent-0.18.1 pytest-7.1.2 pytest-cov-3.0.0 pytest-github-actions-annotate-failures-0.1.7 python-dateutil-2.8.2 pyyaml-6.0 pyzmq-23.2.1 readme-renderer-36.0 requests-toolbelt-0.9.1 rfc3986-2.0.0 rich-12.5.1 ruamel.yaml-0.17.21 ruamel.yaml.clib-0.2.6 scipy-1.9.0 seaborn-0.11.2 setuptools-65.0.2 six-1.16.0 sniffio-1.2.0 soupsieve-2.3.2.post1 sphinx-autodoc-typehints-1.19.2 sphinx-jinja2-compat-0.1.2 sphinx-prompt-1.5.0 sphinx-tabs-3.4.0 sphinx-toolbox-3.2.0 sphinxcontrib-bibtex-2.4.2 stack-data-0.4.0 tabulate-0.8.10 tenacity-8.0.1 terminado-0.15.0 tinycss2-1.1.1 tomli-2.0.1 tomlkit-0.11.4 tornado-6.2 traitlets-5.3.0 twine-4.0.1 typing-extensions-4.3.0 wcwidth-0.2.5 webencodings-0.5.1 websocket-client-1.3.3 wrapt-1.14.1 zipp-3.8.1

In both cases, Python 3.10.4 was used and when we compare the output, also the matplotlib version was obviously the same. However, there was an update from matplotlib-inline-0.1.3 to matplotlib-inline-0.1.5. Is that a possible source?

I know that the project comes with some complexity and that there are a few libraries around that most likely have nothing to do with this error. I will try to work on a minimal example if nothing jumps to our minds here.

I checked the commit between the last successful and first failing build. It only contains some changes in the unittests, no change in dependencies. As both the CI on GitHub and readthedocs are failing, it is rather unlikely that this bug is due to a miraculous specific change on their server side.

[1kastner]

...and tada, at https://github.com/ipython/matplotlib-inline/blob/master/matplotlib_inline/backend_inline.py#L55 we have a return statement that potentially makes the figure manager None. That code changes 3 months ago and released yesterday, see https://pypi.org/project/matplotlib-inline/#history.

[tacaswell]

For now pinning that back will get your tests running again, but someone needs to sort out what the right fix here is.

I was a bit through because you reported using the agg backend in the original post.

I am 95% sure that we need to fix this on the matplotib-inline side.

[tacaswell]

Thank you for tracking this down @1kastner !

[1kastner]

@tacaswell I did not understand matplotlib well enough to tell you that the other backend was effectively used. I am sorry for the confusion I introduced!

[tacaswell]

In general adding print(mpl.get_backend()) right above the offending code will give the right answer.

[jklymak]

We can close this from our end, right?

[tacaswell]

I am only 95% sure it is a bug on their side. It is possible than returning None there used to work and merely opted out of pylab's tracking (rather than never supporting None here and this was a bug introduced in the recent refactor over there).

[1kastner]

I have created a new ticket at their repository and fixed the version in my project for the meantime. With the previous version set, everything works again. I will keep you updated. For me, both options (closing the ticket or waiting for the final answer before doing that) are okay.

[1kastner]

And of course thank you very much for the very fast support!

[jklymak]

Well lets close here and re-open if ipython/matplotlib-inline#18 bounces back to us. Stuff never gets closed otherwise....

[1kastner]

The error was spotted and a new release drafted. Generally speaking, a more detailled error message that also included the currently used backend would have been awesome and it would have saved me from coming here in the first place. But I also see that this is not easy to achieve as matplotlib and its respective backend are deeply integrated.