Only store failed tests mpl-pytest

[cvanelteren]

Pytest does not allow to only store failed tests. This PR addresses this issue by providing a plugin that removes all successful tests from the results dir. It needs to be called as

pytest --mpl --mpl-baseline-path=baseline --mpl-default-style="./ultraplot.yml" --store-failed-only --mpl-results-path=./results ultraplot/tests/test_1dplots.py::test_heatmap_labels --mpl-generate-summary=html

This builds upon the issues raised in #103 #89

[cvanelteren]

Note this reasons from the perspective that all tests are run. It should be better to inverse the pattern (only store failed) but this way is easier and does not require an additional directory.

[cvanelteren]

The idea would be to host the full baseline on a separate repo and push the failed to readthedocs as suggested by @beckermr

[beckermr]

We need to test this live by adding the flag, making a test fail, and ensuring the output looks correct.

[cvanelteren]

Did this locally, but I will push an error now.

[beckermr]

We need to add the flag to the pytest call too in order to test.

[beckermr]

The artifacts are still ~90 MB and I see all of the plots in them. Something is not working here...

[cvanelteren]

Will check later.

[cvanelteren]

Should be good now @beckermr

[cvanelteren]

Locally it would look like:

with the failed test only showing.

[beckermr]

I am trying a failure live again.