Travis error is MacOS image tolerance of 0.005 for test_constrained_layout.py::test_colorbar_location

[jklymak]

Originally posted by @jklymak in #12431

Travis error is MacOS image tolerance of 0.005 changing for test_constrained_layout.py::test_colorbar_location. Not sure what changed on macOS to cause this, but it doesn't affect this PR.

[jklymak]

This seems to have fixed itself 🤷‍♂️

[QuLogic]

I assume you don't know what the result looks like; I just hit this trying to build a package locally, so here's what it looks like:

Note that in this case, the RMS is much higher: 78.990.

[jklymak]

Yeah, its just a flaky test. 90% of the time, if you rerun it, it'll look correct...

I'm guessing the constraint solver gets in a bad loop sometimes or finds an incorrect minima?

Does it really take a long time if this happens for you?

[jklymak]

Also open to suggestions - happy to simplify the test - its a bit of a crazy figure 😉

[jklymak]

i.e. #12678 it failed, and I reran the CI it failed on and it passed...

[QuLogic]

I mean, it was the full test suite, so whether it was slower wasn't particularly noticeable. I also re-ran just the one test 100 times over and it never failed, so...

Should we just mark it as flaky then?