basic testing suite

[krauthex]

the following packages need to be installed for the tests to work:

• fastddm (obviously :) )
• pytest
• pytest-regtest
• py (this one is needed by pytest but in pytests latest version the dependency is missing)

Run the tests then via

pytest tests/test_ddm.py -v

The way this is implemented is the following:

1. run fastddm.ddm once using the cpp core and fft mode & store this result in a pytest fixture
2. run the regression test against the supplied file and check if something changed (for this all other parameters like number of images, lags, etc. have to stay the same; it can be recreated again though with other input data :-) )
3. run all other ddm options using all other cores & modes (ignore the CUDA ones if CUDA is not installed) and check if they produce the same results in terms of the ImageStructureFunction._data contents.

[krauthex]

1 open question: what parts of the ImageStructureFunction should we test, @enrico-lattuada ?

and one other note: as of now, the python version does not return the variance and the average power spectrum in its _data property; so here the tests are slightly adjusted.

I also found out, that when presented with a lag=0, the py-diff version fails because the slicing/cropping of the arrays breaks for lag=0. I should fix this at some point 😅

[krauthex]

still missing is the differentiation between 32 and 64 bit float versions. here I'm not particularly sure yet how we can test this, since there's no way to install both ways together, right? 🤔

[krauthex]

for why the package py needs to be installed, see here.

[krauthex]

just realised that the previous output of the regression test was truncated (because the default print function truncates rows and columns of numpy arrays if they are 'too big' to be displayed), so for exactly this print statement I set the truncation limit higher; now the whole array is printed, unfortunately it's 11MB big 😁

in any case, to re-generate the regression file and make the regression test pass again, one can run

pytest tests/test_ddm.py --regtest-reset

[enrico-lattuada]

1 open question: what parts of the ImageStructureFunction should we test, @enrico-lattuada ?

I think just the import is ok for now. When we finalize the thing about the save/load within the pims framework, it might be worth testing also this functionality (we need to be sure that the endianness is correctly taken into account when writing/reading binary files).

[enrico-lattuada]

still missing is the differentiation between 32 and 64 bit float versions. here I'm not particularly sure yet how we can test this, since there's no way to install both ways together, right? thinking

Correct. And also in this case, we may want to switch to a single-precision-output only at some point, since the double precision output is not really needed most of the time.

[krauthex]

1 open question: what parts of the ImageStructureFunction should we test, @enrico-lattuada ?

I think just the import is ok for now. When we finalize the thing about the save/load within the pims framework, it might be worth testing also this functionality (we need to be sure that the endianness is correctly taken into account when writing/reading binary files).

I mean in the test_ddm.py the contents of the ._data or of the .data property are tested, so not only the imports ;)

[krauthex]

still missing is the differentiation between 32 and 64 bit float versions. here I'm not particularly sure yet how we can test this, since there's no way to install both ways together, right? thinking

Correct. And also in this case, we may want to switch to a single-precision-output only at some point, since the double precision output is not really needed most of the time.

ah single precision output will help here .. still not sure what a good way would be to simply test this 🤔 maybe pytest supports testing multiple versions of a software; I have to check. I do remember that in the past I used a tool called tox that was capable of testing multiple python versions.. maybe this is also something for us?

[enrico-lattuada]

1 open question: what parts of the ImageStructureFunction should we test, @enrico-lattuada ?

I think just the import is ok for now. When we finalize the thing about the save/load within the pims framework, it might be worth testing also this functionality (we need to be sure that the endianness is correctly taken into account when writing/reading binary files).

I mean in the test_ddm.py the contents of the ._data or of the .data property are tested, so not only the imports ;)

Sorry. Yes, other class methods are not that complex, so I think we can decide later if we want some other class features to be tested.

[enrico-lattuada]

still missing is the differentiation between 32 and 64 bit float versions. here I'm not particularly sure yet how we can test this, since there's no way to install both ways together, right? thinking

Correct. And also in this case, we may want to switch to a single-precision-output only at some point, since the double precision output is not really needed most of the time.

ah single precision output will help here .. still not sure what a good way would be to simply test this thinking maybe pytest supports testing multiple versions of a software; I have to check. I do remember that in the past I used a tool called tox that was capable of testing multiple python versions.. maybe this is also something for us?

We can check, sure

[krauthex]

testing the 32-bit implementations can wait for now, since we do not plan to put them into the first official release!

[krauthex]

all backends & modes now produce the same results with fastddm.ddm; the py core needed the additional average image power spectrum and the variance, and the py diff mode still had an issue with a lag=0.