Backend switching

[anntzer]

This is the followup to #9551 (backend loading refactor), and includes it (so it would make sense for #9551 to be merged first).

This PR removes the ambiguous behavior of matplotlib.use, which was, up to now, documented as follows:

This function must be called before importing pyplot for the first time; or, if you are not using pyplot, it must be called before importing matplotlib.backends. If warn is True, a warning is issued if you try and call this after pylab or pyplot have been loaded. "In certain black magic use cases, e.g. pyplot.switch_backend, we are doing the reloading necessary to make the backend switch work (in some cases, e.g., pure image backends) so one can set warn=False to suppress the warnings.)

Instead, matplotlib.use now works just fine as long as an event loop has not started yet (i.e. an interactive window popped up), or when switching to a noninteractive backend, or to an interactive backend compatible with the current event loop (e.g. gtk3agg -> gtk3cairo); in all other cases, an exception is raised. This means, for example, that one can write from matplotlib import pyplot as plt; plt.rcParams["backend"] = "foo"; ... instead of having to do a separate import of matplotlib.

Moreover, it is now possible to set rcParams["backend"] to a list of candidate backends; they will simply be tested one at a time until one succeeds. If pyplot is already loaded, the value in rcParams["backend"] is immediately replaced by the successful candidate. If pyplot is not loaded yet, rcParams["backend"] will keep hold of the list of candidates until pyplot is loaded. This is a documented change of behavior (users should import pyplot to force backend resolution). (Alternatively the list could be shoved into a new, separate rcParam, but little would be gained because rcParams["backend"] would then hold an incorrect value until backend resolution). Another change of behavior is that the matplotlib.backends.backend global variable has been removed, again in favor of rcParams["backend"] as single truth point: it was too difficult to properly keep matplotlib.backends.backend in sync, especially at import time when that module may not have been imported yet.

(Exception: I don't have a Mac available to figure out the relevant fixes for the OSX backend, but hopefully another contributor can help here. See _get_current_event_loop.)

Edit: Now also working on OSX, see comment below.

A major advantage of such a scheme is that we can now just ship a default matplotlibrc which lists all buitin backends in preference order (backend: osx, qt5agg, qt4agg, gtk3agg, ...). Currently, this list is evaluated at build time (by testing which GUI toolkit is importable), which causes two major problems: 1) the import of the GUI toolkits need to happen in separate processes, which means that setup.py relies on multiprocessing(!); for example, this appears to confuse conda-build (see comment in ci/conda_recipe/meta.yaml); 2) the built list is useless anyways unless 1) the end user is doing the build, rather than a central packager (the end user may have a different list of installed GUI toolkits...) and 2) the end user never adds or removes a new GUI toolkit. Here, I can instead completely strip out the relevant part of setupext.py, greatly simplifying it.

The backend_fallback rcParam likewise becomes obsolete, although this PR does not remove it (this rcParam can easily go through a normal deprecation route).

I also removed the interactive backends API docs because the shimming (to allow them to be imported in a headless buildbot) was insufficient to support this PR's implementation. Proposed fix at #9708 (comment). (But note that they were incomplete anyways...)

Closes #3679, #6739, #8613, #9017 (comment); sort-of #3466 (which is perhaps more a doc issue...).

Some comments to try clarifying the implementation:

• matplotlib.use only supports a single argument (not a list of candidate backends) because we want to keep use(...); rcdefaults() working: this implies that use(...) must overwrite the default backend (in rcParamsDefault, and likewise rcParamsOrig), to a single fixed value.
• setting multiple entries in rcParams["setup"] defers to pyplot.switch_backends, which itself defers to backends.pylab_setup. Once the correct candidate is found, backends.pylab_setup sets the single entry in rcParams["setup"].

[jklymak]

This looks really cool. But, I'm on MacOS so its not working for me yet. If I do

import matplotlib
print(matplotlib.get_backend())
matplotlib.use('Qt5Agg')
import matplotlib.pyplot as plt
plt.scatter([1, 2], [0,2])
print(matplotlib.get_backend())
plt.show()

I get a Qt5Agg GUI and

['MacOSX', 'Qt5Agg', 'Qt4Agg', 'GTK3Agg', 'GTK3Cairo', 'GTKAgg', 'TkAgg', 'WXAgg', 'agg', 'cairo']
Qt5Agg

If I do

import matplotlib
print(matplotlib.get_backend())
import matplotlib.pyplot as plt
matplotlib.use('Qt5Agg')
plt.scatter([1, 2], [0,2])
print(matplotlib.get_backend())
plt.show()

I get:

['MacOSX', 'Qt5Agg', 'Qt4Agg', 'GTK3Agg', 'GTK3Cairo', 'GTKAgg', 'TkAgg', 'WXAgg', 'agg', 'cairo']
Qt5Agg

but, I get the macOS GUI (Toolbar on the bottom instead of top, and I can close with command-w). So either I am not understanding how to test this or MacOS needs work.

Happy to help make this work on MacOS...

[anntzer]

That's because I forgot to actually update the backend when doing rcParams["backend"] = "foo" (as opposed to use()), should hopefully be fixed now (if I didn't break something else in the meantime, but at least the example you posted works for me replacing OSX/Qt5 by Gtk3/Qt5).

Right now what needs to be done for OSX is a way to detect whether the OSX event loop is already running (see _get_current_event_loop for examples), so that having an OSX event loop running prevents starting another event loop.

[jklymak]

Confirmed, the snippet above now works as advertized!

[jklymak]

BTW, looking at event-loops and such would be a longer term project for me if I were to help. @dopplershift would probably be much better suited.

[anntzer]

I just tried on your machine and it worked for me... Can you check what's happening at INFO log level, especially during the call to matplotlib.backends.pylab_setup?

[jklymak]

@anntzer is this still active? Sorry if I was supposed to get back to you on some of it - looks like it'll be an ugly rebase, but they are probably all your changes anyways 😉

[anntzer]

I'll probably wait until the rest of the mpl3 transition settles down a bit before revisiting this.

[tacaswell]

Some version of this should go in for 3.0 to make life easier for conda-forge.

[anntzer]

Frankly, I wouldn't say no to some progress on merging my 61 other currently open PRs rather than seeing them bitrot one after another like this one.

[tacaswell]

Closing in favor of #11519, #11520, and #11600