[Bug]: spurious test failures

[MeggyCal]

Bug summary

Hi, thank you for maintaining this amazing library! I am a packager in openSUSE. During the development process of openSUSE Tumbleweed we periodically rebuild the whole (future) distribution and we noticed some test failures which look... well, odd.

First, while trying to update gc to 8.2.10, test_backend_svg.py::test_rasterized[svg] failed on image comparison. Sadly, I don't have that images.

Second, while trying to update xkeyboard-config to 2.46, test_getattr.py::test_getattr with any gtk3-related backend hangs and is killed after almost two hours for being idle.

It is puzzling for me (I don't see what the packages have in common with matplotlib), but the same result is produced after multiple rebuilds, so it isn't just random failure. Also see https://bugzilla.suse.com/show_bug.cgi?id=1252721. Could you please take a look into this issue?

A bit of a mystery... well, it is Halloween after all 🙂 .

Code for reproduction

run tests, once with gc 8.2.10, for the second time with xkeyboard-config 2.46

Actual outcome

gc:

[  447s] >           _raise_on_image_difference(
[  447s]                 ref_image_path, test_image_path, tol=tol
[  447s]             )
[  447s] E           matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 147.627):
[  447s] E           	result_images/test_backend_svg/test_rasterized[svg]_svg.png
[  447s] E           	result_images/test_backend_svg/test_rasterized[svg]-expected_svg.png
[  447s] E           	result_images/test_backend_svg/test_rasterized[svg]_svg-failed-diff.png
[  447s] 
[  447s] /usr/lib64/python3.11/site-packages/matplotlib/testing/decorators.py:422: ImageComparisonFailure

xkeyboard-config:

[  547s] [gw3] [ 94%] PASSED test_getattr.py::test_getattr[matplotlib.backends.backend_cairo] <- ../../../../../../usr/lib64/python3.11/site-packages/matplotlib/tests/test_getattr.py 
[ 5957s] test_getattr.py::test_getattr[matplotlib.backends.backend_gtk3] <- ../../../../../../usr/lib64/python3.11/site-packages/matplotlib/tests/test_getattr.py qemu-kvm: terminating on signal 15 from pid 68319 ()

Expected outcome

tests should pass

Additional information

this has worked with earlier versions of gc and xkeyboard-config

Operating system

openSUSE Tumbleweed

Matplotlib Version

3.10.7

Matplotlib Backend

differs, gtk3 version 3.24.51

Python version

3.11.14

Jupyter version

none

Installation

from source (.tar.gz)

[timhoffm]

Concerning the test_getattr failure, that test is basically only doing import matplotlib.backends.backend_gtk3. I suppose GTK3 somehow gets tripped up by xkeyboard-config. Can you try running this python import directly in a python environment in the context of the new xkeyboard-config version?

[QuLogic]

First, while trying to update gc to 8.2.10, test_backend_svg.py::test_rasterized[svg] failed on image comparison. Sadly, I don't have that images.

What is gc? The garbage collector? I doubt it'd be an intentional garbage collector change and anyway, we don't use it ourselves. That RMS failure is so large, it's either a bug in gc or Inkscape.

Second, while trying to update xkeyboard-config to 2.46, test_getattr.py::test_getattr with any gtk3-related backend hangs and is killed after almost two hours for being idle.

You may want to try #30533, which moves those tests into subprocesses with a much shorter timeout.

[QuLogic]

In fact, given the large RMS, I suspect that the rasterized image is probably missing entirely. You should definitely try to download the results and check. And for Inkscape, that's probably the gdk-pixbuf2 change to the glycin loaders, though it was fixed recently somehow.

See, e.g., https://bugzilla.redhat.com/show_bug.cgi?id=2389647

[MeggyCal]

Hi, thank you for the quick answers, even provididng solutions! That is a lot to do, I will try to get to it as soon as possible.

[MeggyCal]

Concerning the test_getattr failure, that test is basically only doing import matplotlib.backends.backend_gtk3. I suppose GTK3 somehow gets tripped up by xkeyboard-config. Can you try running this python import directly in a python environment in the context of the new xkeyboard-config version?

python3 -c 'import matplotlib.backends.backend_gtk3' succeeded in the same environment.

[MeggyCal]

After timeouting the tests hanging with xkeyboard-config the testsuite finished and revealed more timeouts in test_determinism_check and test_determinism_source_date_epoch, all of them failing with:

subprocess.TimeoutExpired: Command '['/usr/bin/python3.11', '-R', '-c', "from matplotlib.tests.test_determinism import _save_figure; _save_figure(...)"]' timed out after 60 seconds

List of timeouted tests:

[  577s] FAILED test_determinism.py::test_determinism_check[p-svg-False]
[  577s] FAILED test_determinism.py::test_determinism_check[mhip-svg-False]
[  577s] FAILED test_determinism.py::test_determinism_check[mhip-svg-True]
[  577s] FAILED test_determinism.py::test_determinism_source_date_epoch[pdf-/CreationDate (D:20000101000000Z)]
[  577s] FAILED test_determinism.py::test_determinism_source_date_epoch[ps-%%CreationDate: Sat Jan 01 00:00:00 2000]

[timhoffm]

We have been struggling with flaky timeout behavior for some time. See #29797. At some point we thought, we had fixed it though updating the runner, but it sometimes happens again. So it's not eventually solved. Possibly we have to revive #29981 and/or dig deeper.