image compare animations for testing

[t-makaro]

I'm currently trying to create pytests for animations over in animatplot. Now, I know that people here have talked about writing image comparison tests for animations, so I thought I would open an issue here to track progress, and to share what I am doing.

Now my needs for animatplot are fairly different, but essentially I'm going to wrap around matplotlib.testing.compare.compare_images, and use a custom file writer (which you can see here) to output a set of images. This file writer was inspired by @tacaswell's stackoverflow answer.

Once I get stuff working over there, I will be willing to help create these types of tests here.

[tacaswell]

https://github.com/matplotlib/pytest-mpl might also be interested in this feature.

[t-makaro]

I'm very close to having everything working, but I'm having to remove the text to get things to compare properly on circleci. Which is fine for the most part, but for one of my tests I can't remove the text. I am getting a:

Could not set locale to English/United States. Some date-related tests may fail
    "Could not set locale to English/United States. "

message, but I don't think that is the issue.

I'm not sure what's causing the test to render so differently.

I am using matplotlib.testing.setup() which should set the font....

[tacaswell]

Might be a freetype issue, see https://matplotlib.org/devel/contributing.html#building-matplotlib-for-image-comparison-tests

[t-makaro]

Alright, I have something that works for me over at animatplot. You can see my implementation here. It's as simple as using a decorator on a function that returns an animation. Internally, it will use the BunchOFiles writer to save the animation, and then loop over the images for comparison. Unfortunately, it does require specifying how many frames to expect.

This would require some additional work to suit matplotlib's needs.