Qt5 test auto-skip is not working correctly

[QuLogic]

When both Qt4 and Qt5 are available, then the automatic skip for Qt-specific tests does not seem to work correctly. This probably cannot be triggered in conda, because you can only have one version installed, but it is possible and a problem when building distro packages.

On v3.1.x (ebb7d28), Qt5Agg is automatically picked and Qt4Agg is skipped:

$ python3 tests.py -v matplotlib.tests.test_backend_qt -k 'not test_dpi_ratio_change'
Python byte-compilation optimization level: 0
=========================================================== test session starts ============================================================
platform linux -- Python 3.7.3, pytest-3.6.4, py-1.5.4, pluggy-0.6.0
rootdir: /home/elliott/code/matplotlib, inifile: pytest.ini
plugins: xdist-1.22.5, tornado-0.7.0, forked-0.2, cov-2.5.1
collected 28 items / 1 deselected                                                                                                          

lib/matplotlib/tests/test_backend_qt.py::test_fig_close[Qt4Agg] SKIPPED                                                              [  3%]
lib/matplotlib/tests/test_backend_qt.py::test_fig_close[Qt5Agg] PASSED                                                               [  7%]
lib/matplotlib/tests/test_backend_qt.py::test_fig_signals PASSED                                                                     [ 11%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-shift] SKIPPED                                                      [ 14%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-lower] SKIPPED                                                      [ 18%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-control] SKIPPED                                                    [ 22%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-unicode_upper] SKIPPED                                              [ 25%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-unicode_lower] SKIPPED                                              [ 29%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-alt_control] SKIPPED                                                [ 33%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-control_alt] SKIPPED                                                [ 37%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-modifier_order] SKIPPED                                             [ 40%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-backspace] SKIPPED                                                  [ 44%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-backspace_mod] SKIPPED                                              [ 48%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-non_unicode_key] SKIPPED                                            [ 51%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-shift] PASSED                                                       [ 55%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-lower] PASSED                                                       [ 59%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-control] PASSED                                                     [ 62%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-unicode_upper] PASSED                                               [ 66%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-unicode_lower] PASSED                                               [ 70%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-alt_control] PASSED                                                 [ 74%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-control_alt] PASSED                                                 [ 77%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-modifier_order] PASSED                                              [ 81%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-backspace] PASSED                                                   [ 85%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-backspace_mod] PASSED                                               [ 88%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-non_unicode_key] PASSED                                             [ 92%]
lib/matplotlib/tests/test_backend_qt.py::test_subplottool PASSED                                                                     [ 96%]
lib/matplotlib/tests/test_backend_qt.py::test_figureoptions PASSED                                                                   [100%]

=========================================== 15 passed, 12 skipped, 1 deselected in 0.49 seconds ============================================

But if Qt4 is loaded already (say, by setting MPLBACKEND or in matplotlibrc), then Qt5Agg errors out instead of skipping:

$ MPLBACKEND=Qt4Agg python3 tests.py -v matplotlib.tests.test_backend_qt -k 'not test_dpi_ratio_change'
Python byte-compilation optimization level: 0
=========================================================== test session starts ============================================================
platform linux -- Python 3.7.3, pytest-3.6.4, py-1.5.4, pluggy-0.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/elliott/code/matplotlib, inifile: pytest.ini
plugins: xdist-1.22.5, tornado-0.7.0, forked-0.2, cov-2.5.1
collected 28 items / 1 deselected                                                                                                          

lib/matplotlib/tests/test_backend_qt.py::test_fig_close[Qt4Agg] PASSED                                                               [  3%]
lib/matplotlib/tests/test_backend_qt.py::test_fig_close[Qt5Agg] ERROR                                                                [  7%]
lib/matplotlib/tests/test_backend_qt.py::test_fig_signals ERROR                                                                      [ 11%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-shift] PASSED                                                       [ 14%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-lower] PASSED                                                       [ 18%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-control] PASSED                                                     [ 22%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-unicode_upper] PASSED                                               [ 25%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-unicode_lower] PASSED                                               [ 29%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-alt_control] PASSED                                                 [ 33%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-control_alt] PASSED                                                 [ 37%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-modifier_order] PASSED                                              [ 40%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-backspace] PASSED                                                   [ 44%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-backspace_mod] PASSED                                               [ 48%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-non_unicode_key] PASSED                                             [ 51%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-shift] ERROR                                                        [ 55%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-lower] ERROR                                                        [ 59%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-control] ERROR                                                      [ 62%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-unicode_upper] ERROR                                                [ 66%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-unicode_lower] ERROR                                                [ 70%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-alt_control] ERROR                                                  [ 74%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-control_alt] ERROR                                                  [ 77%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-modifier_order] ERROR                                               [ 81%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-backspace] ERROR                                                    [ 85%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-backspace_mod] ERROR                                                [ 88%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-non_unicode_key] ERROR                                              [ 92%]
lib/matplotlib/tests/test_backend_qt.py::test_subplottool ERROR                                                                      [ 96%]
lib/matplotlib/tests/test_backend_qt.py::test_figureoptions ERROR                                                                    [100%]

request = <SubRequest 'mpl_test_settings' for <Function 'test_fig_close[Qt5Agg]'>>

    @pytest.fixture(autouse=True)
    def mpl_test_settings(request):
        from matplotlib.testing.decorators import _cleanup_cm
    
        with _cleanup_cm():
    
            backend = None
            backend_marker = request.node.get_closest_marker('backend')
            if backend_marker is not None:
                assert len(backend_marker.args) == 1, \
                    "Marker 'backend' must specify 1 backend."
                backend, = backend_marker.args
                prev_backend = matplotlib.get_backend()
    
            style = '_classic_test'  # Default of cleanup and image_comparison too.
            style_marker = request.node.get_closest_marker('style')
            if style_marker is not None:
                assert len(style_marker.args) == 1, \
                    "Marker 'style' must specify 1 style."
                style, = style_marker.args
    
            matplotlib.testing.setup()
            if backend is not None:
                # This import must come after setup() so it doesn't load the
                # default backend prematurely.
                import matplotlib.pyplot as plt
                try:
>                   plt.switch_backend(backend)

lib/matplotlib/testing/conftest.py:44: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

newbackend = 'Qt5Agg'

    def switch_backend(newbackend):
        """
        Close all open figures and set the Matplotlib backend.
    
        The argument is case-insensitive.  Switching to an interactive backend is
        possible only if no event loop for another interactive backend has started.
        Switching to and from non-interactive backends is always possible.
    
        Parameters
        ----------
        newbackend : str
            The name of the backend to use.
        """
        close("all")
    
        if newbackend is rcsetup._auto_backend_sentinel:
            # Don't try to fallback on the cairo-based backends as they each have
            # an additional dependency (pycairo) over the agg-based backend, and
            # are of worse quality.
            for candidate in ["macosx", "qt5agg", "qt4agg", "gtk3agg", "tkagg",
                              "wxagg", "agg"]:
                try:
                    switch_backend(candidate)
                except ImportError:
                    continue
                else:
                    rcParamsOrig['backend'] = candidate
                    return
    
        backend_name = (
            newbackend[9:] if newbackend.startswith("module://")
            else "matplotlib.backends.backend_{}".format(newbackend.lower()))
    
        backend_mod = importlib.import_module(backend_name)
        Backend = type(
            "Backend", (matplotlib.backends._Backend,), vars(backend_mod))
        _log.debug("Loaded backend %s version %s.",
                   newbackend, Backend.backend_version)
    
        required_framework = Backend.required_interactive_framework
        if required_framework is not None:
            current_framework = \
                matplotlib.backends._get_running_interactive_framework()
            if (current_framework and required_framework
                    and current_framework != required_framework):
                raise ImportError(
                    "Cannot load backend {!r} which requires the {!r} interactive "
                    "framework, as {!r} is currently running".format(
>                       newbackend, required_framework, current_framework))
E               ImportError: Cannot load backend 'Qt5Agg' which requires the 'qt5' interactive framework, as 'qt4' is currently running

lib/matplotlib/pyplot.py:230: ImportError

Even stranger, if you set the backend to Qt5Agg, then the Qt4 tests still run:

$ MPLBACKEND=Qt5Agg python3 tests.py -v matplotlib.tests.test_backend_qt -k 'not test_dpi_ratio_change'
Python byte-compilation optimization level: 0
=========================================================== test session starts ============================================================
platform linux -- Python 3.7.3, pytest-3.6.4, py-1.5.4, pluggy-0.6.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /home/elliott/code/matplotlib, inifile: pytest.ini
plugins: xdist-1.22.5, tornado-0.7.0, forked-0.2, cov-2.5.1
collected 28 items / 1 deselected                                                                                                          

lib/matplotlib/tests/test_backend_qt.py::test_fig_close[Qt4Agg] PASSED                                                               [  3%]
lib/matplotlib/tests/test_backend_qt.py::test_fig_close[Qt5Agg] ERROR                                                                [  7%]
lib/matplotlib/tests/test_backend_qt.py::test_fig_signals ERROR                                                                      [ 11%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-shift] PASSED                                                       [ 14%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-lower] PASSED                                                       [ 18%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-control] PASSED                                                     [ 22%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-unicode_upper] PASSED                                               [ 25%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-unicode_lower] PASSED                                               [ 29%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-alt_control] PASSED                                                 [ 33%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-control_alt] PASSED                                                 [ 37%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-modifier_order] PASSED                                              [ 40%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-backspace] PASSED                                                   [ 44%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-backspace_mod] PASSED                                               [ 48%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt4Agg-non_unicode_key] PASSED                                             [ 51%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-shift] ERROR                                                        [ 55%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-lower] ERROR                                                        [ 59%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-control] ERROR                                                      [ 62%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-unicode_upper] ERROR                                                [ 66%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-unicode_lower] ERROR                                                [ 70%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-alt_control] ERROR                                                  [ 74%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-control_alt] ERROR                                                  [ 77%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-modifier_order] ERROR                                               [ 81%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-backspace] ERROR                                                    [ 85%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-backspace_mod] ERROR                                                [ 88%]
lib/matplotlib/tests/test_backend_qt.py::test_correct_key[Qt5Agg-non_unicode_key] ERROR                                              [ 92%]
lib/matplotlib/tests/test_backend_qt.py::test_subplottool ERROR                                                                      [ 96%]
lib/matplotlib/tests/test_backend_qt.py::test_figureoptions ERROR                                                                    [100%]

[tacaswell]

I am going to do an rc2 and then look at this.

[tacaswell]

It seems that just importing the bindings is not enough to fail:

(mpl_test) ✘  ~/src/p/matplotlib [v3.1.x ✭|⚑ 3]
sharon@23:17  ➤  ipython
imporPython 3.6.8 (default, Apr  9 2019, 04:59:38) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import PyQt4                                                                                                                                                                                                                                          

In [2]: import PyQt5                                                                                                                                                                                                                                          

In [3]:                                                                                                                                                                                                                                                       
Do you really want to exit ([y]/n)? y
(mpl_test) ✔  ~/src/p/matplotlib [v3.1.x ✭|⚑ 3]
sharon@23:19  ➤  ipython
Python 3.6.8 (default, Apr  9 2019, 04:59:38) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import PyQt5                                                                                                                                                                                                                                          

In [2]: import PyQt4                                                                                                                                                                                                                                          

In [3]:        

I remember this used to fail at import time? Either my memory is off (very possible) or riverbank changed things so they don't conflict as badly at import time (also very possible).

[vasole]

@tacaswell

Our experience is that one should test for XXXX.QtCore to make sure the Qt binding XXXX is actually imported (see #14153)

[tacaswell]

Lets see if #14193 will fix this.