Output figure diff in test figure page

[dstansby]

Requires remote data to download the baseline images to generate a diff, but I'm not sure if there's a way to avoid that?

[pep8speaks]

Hello @dstansby! Thanks for updating the PR.

• There are no PEP8 issues in the file sunpy/conftest.py !

• There are no PEP8 issues in the file sunpy/tests/helpers.py !

Comment last updated on October 01, 2018 at 19:50 Hours UTC

[ghost]

Thanks for the pull request @dstansby! Everything looks great!

[nabobalis]

How would you feel about switching the other of the plots? Either old, new, diff or old, diff, new?

[dstansby]

Happy to change the order, maybe old diff new works best.

[dstansby]

As it stands in this PR doing python setup.py test --figure-only won't do anything because it now needs --online to get the baseline images, maybe I should make generating the diff contingent on --onilne being passed so figure tests can still be run offline?

[nabobalis]

Yes, I do think we want to be able to at least run the figure tests offline.

[dstansby]

@Cadair is there a way to work out when running the tests if the --online option has been passed to pytest? I want to only do the image comparison if that's the case.

[Cadair]

@dstansby I mean pytest will know, where do you want to do the check?

[Cadair]

@dstansby I worked out how to do it:

def pytest_unconfigure(config):
    try:
        rd = config.getoption('remote_data')
    except ValueError:
        rd = 'none'
    remotedata = rd == 'any'

However, I think it's silly that we have to specify --online to be able to generate the image comparisons, so I hacked in a way to enable it again, see PR dstansby#1

[Cadair]

A general question about this: What happens if the baseline image hasn't been uploaded to the repo yet? It looks like in that case it might explode in quite a few places?

[dstansby]

I've added some code to handle baseline images not being available.

[Cadair]

I don't think this is needed any more.

[Cadair]

I think this is basically good to go. It would be good to have a test or two for the HTML page comparison function?

[Cadair]

oh also a changelog

[dstansby]

@Cadair do your changes mean that the --online flag isn't really respected now? (ie. if you don't pass --online, running tests will still connect to the internet to do the figure comparison)

[Cadair]

Yes, it will now connect to the internet irrespective of the online flag. Specifically for building the comparison page. (So the building should fail gracefully if the machine is offline)

[dstansby]

Hmm, but that seems a bit dishonest, if I don't pass the --online flag then I would expect my machine not to try and connect to the internet. There must be a way to detect if the --online flag has been passed as a way of deciding whether to fetch the baseline images.

[Cadair]

The idea of the --online flag is really to run the tests which are marked as such. I don't really consider the generation of the summary page part of the test run. Also, I don't want to have to run the online tests just to build the figure comparison page (i.e. I might just want to see the comparisons for offline tests).

Can we not just have the comparison builder attempt to download the images and fail gracefully, is this not the same effect as respecting --online without entangling ourselves with the selection of the tests to run?

[dstansby]

👍 okay, I'm happy for this to go in as is. This is a gentle nudge that now Matplotlib 3.0 has just been released it would be nice to have this PR merged so we can see the differences in Sunpy test images with the new release.

[nabobalis]

@dstansby Any chance of a changelog entry? I think this PR does warrant one.

Also do we want to still add --online to the CI command?