Plotting/cairo-related test failures

[szhorvat]

I'm getting test failures on main when running the test suite on my own computer. This is with the latest commit as of today, 4db1260

Any idea what is going wrong here, and why it doesn't fail on the CI?

I have:

~/R/python-igraph (main|✔) [2] $ conda list cairo
# packages in environment at /Users/szhorvat/opt/miniconda3/envs/igdev:
#
# Name                    Version                   Build  Channel
cairo                     1.16.0               h3ce6f7e_4  
pycairo                   1.23.0          py310h0eaaf77_0  
~/R/python-igraph (main|✔) $ conda list matplotlib
# packages in environment at /Users/szhorvat/opt/miniconda3/envs/igdev:
#
# Name                    Version                   Build  Channel
matplotlib                3.7.1           py310hecd8cb5_1  
matplotlib-base           3.7.1           py310ha533b9c_1  
matplotlib-inline         0.1.6           py310hecd8cb5_0  

Python 3.10.11 on macOS 10.14.6.

---

Failures:

$ python -m unittest                                                                                                       (igdev) 
.FFFFF...............................................3 0
...................................................................................s...............................................................................................ss.....s..s...s............s...........sss....................................................................................................................................................................................
======================================================================
FAIL: test_clustering_directed_small (tests.drawing.cairo.test_graph.ClusteringTestRunner)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/szhorvat/Repos/python-igraph/tests/drawing/cairo/utils.py", line 131, in wrapper
    self.assertLessEqual(res, tol)
AssertionError: 7973492 not less than or equal to 0

======================================================================
FAIL: test_basic (tests.drawing.cairo.test_graph.GraphTestRunner)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/szhorvat/Repos/python-igraph/tests/drawing/cairo/utils.py", line 131, in wrapper
    self.assertLessEqual(res, tol)
AssertionError: 3630634 not less than or equal to 0

======================================================================
FAIL: test_directed (tests.drawing.cairo.test_graph.GraphTestRunner)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/szhorvat/Repos/python-igraph/tests/drawing/cairo/utils.py", line 131, in wrapper
    self.assertLessEqual(res, tol)
AssertionError: 3785661 not less than or equal to 0

======================================================================
FAIL: test_mark_groups (tests.drawing.cairo.test_graph.GraphTestRunner)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/szhorvat/Repos/python-igraph/tests/drawing/cairo/utils.py", line 131, in wrapper
    self.assertLessEqual(res, tol)
AssertionError: 3785661 not less than or equal to 0

======================================================================
FAIL: test_mark_groups_squares (tests.drawing.cairo.test_graph.GraphTestRunner)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/szhorvat/Repos/python-igraph/tests/drawing/cairo/utils.py", line 131, in wrapper
    self.assertLessEqual(res, tol)
AssertionError: 3901834 not less than or equal to 0

----------------------------------------------------------------------
Ran 454 tests in 3.850s

FAILED (failures=5, skipped=10)

[iosonofabio]

Works on my machine on the same commit...

[szhorvat]

I bisected the range, problem was introduced in f01cb42 Ping @ntamas

f01cb4202420111a145e9cf97e12263839804344 is the first bad commit
commit f01cb4202420111a145e9cf97e12263839804344
Author: Tamas Nepusz <ntamas@gmail.com>
Date:   Thu Mar 23 10:44:51 2023 +0100

    test: make Cairo drawing tests deterministic

 .../cairo/baseline_images/clustering_directed.png   | Bin 31493 -> 33794 bytes
 tests/drawing/cairo/baseline_images/graph_basic.png | Bin 16701 -> 18287 bytes
 .../cairo/baseline_images/graph_directed.png        | Bin 17353 -> 19005 bytes
 .../baseline_images/graph_mark_groups_directed.png  | Bin 17353 -> 19005 bytes
 .../graph_mark_groups_squares_directed.png          | Bin 14924 -> 16548 bytes
 tests/drawing/cairo/test_graph.py                   |  10 +++++++---
 6 files changed, 7 insertions(+), 3 deletions(-)

[ntamas]

I used to get similar test failures before commit f01cb42, that's why I changed the expected images to the ones that I observed on my machine to fix the tests :) So apparently the pixel-by-pixel comparison of the expected and the observed images on your machine is too high. It would be worth overlaying the two images (the expected one and the one you got) on top of each other to see what the differences are.

[szhorvat]

Do you have the same cairo and pycairo versions?

[szhorvat]

I get this for clustering_directed.png:

It is rotated compared to the baseline. This can't be due to Cairo. The computed layout must be different.

Overlay screenshot:

[szhorvat]

I'm pretty sure I made a mistake here by not setting the seed correctly. @ntamas, where are the images saved when the test suite is run as normal? Are they retrievable?

Wouldn't it be better to set a seed separately for each and every test, instead of at the beginning of tests? It's unclear to me when and how GraphTestRunner.setUp is run, as I'm not sufficiently familiar with Python testing.

FYI I ran the tests with python -m unittest. Could that be a source of difference?

[ntamas]

Search for seed in the tests/ folder -- you can see that seeds are set on a per-test basis, not on the level of the entire test suite. The setUp methods are run once before every test in the class where it's defined.

The baseline images are in tests/drawing/*/baseline_images.

[szhorvat]

The baseline images are in tests/drawing/*/baseline_images.

What I was looking for was where the actual images are saved, and whether I can retrieve them for comparison. I set a seed and ran the code from the test manually, like so. There is still a rotation mismatch.

Python 3.10.11 (main, Apr 20 2023, 13:59:00) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import random
>>> from igraph import *
>>> random.seed(42)
>>> g = Graph.Ring(5, directed=True)
>>> clu = VertexClustering(g, [0] * 5)
>>> plot(clu, backend='cairo', target='clustering_directed.png', mark_groups=True)
<igraph.drawing.cairo.plot.CairoPlot object at 0x109072d10>
>>> exit()

[szhorvat]

@ntamas, with tox the tests pass, with python -m unittest they don't. Are you sure these two systems handle the seeds in the same way?

[ntamas]

Conclusion was that the Cairo tests were not running in CI or with tox at all so that's why we haven't seen the failures there. Now the test setup is fixed so Cairo tests are being executed both in CI (where possible) and also when doing a clean install using tox. I've also updated the baseline images.