New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add function to fill acceptance image from curve #248

Merged
merged 4 commits into from Apr 9, 2015

Conversation

Projects
None yet
2 participants
@mapazarr
Member

mapazarr commented Mar 27, 2015

This pull request addresses issue #5.

The method is slow (loop over pixels for calculating offset and interpolate the acceptance value), so maybe it could still be improved.

I still am thinking about a good test for the method. While developping the method I used a script that plots the radial acceptance image that I can see, but I am not sure how to best implement an automatic test on a method working on images. In addition, since the mthod is slow, the test might take longer than desirable. Suggestions are welcome!

@mapazarr

This comment has been minimized.

Show comment
Hide comment
@mapazarr

mapazarr Mar 27, 2015

Member

I checked the output from travis: the failed status seems to come from an error in the connection to the repository.

Member

mapazarr commented Mar 27, 2015

I checked the output from travis: the failed status seems to come from an error in the connection to the repository.

Show outdated Hide outdated gammapy/background/fov.py Outdated
Show outdated Hide outdated gammapy/background/fov.py Outdated
Show outdated Hide outdated gammapy/background/fov.py Outdated
Show outdated Hide outdated gammapy/background/fov.py Outdated
Show outdated Hide outdated gammapy/background/fov.py Outdated
@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Mar 27, 2015

Member

@mapazarr - Thanks!

As you say, a test needs to be added (do this first) and the function needs to be re-written so that it's fast (do this second).

Some suggestions for the test:

  • Use make_empty_image to get an input image.
  • Create some offset and acceptance objects from scratch, or add a utliity function to make it easy to create them for testing (see example here where I've done something similar).

To make it fast, you have to avoid the for loop in Python and instead use numpy arrays.
Use coordinates to get 2D coordinate images and then from that compute a 2D offset image and then evaluate the acceptance curve. For that last part you have to use interpolation, I think, have a look here.

Some info about pyfact.py: it's old spagetti code with useful functionality, but it has to be re-written or re-factored completely and tests / docs added ... this will be a large part of your work ... in most cases it will be simpler if you start from scratch and we just remove pyfact.py at some point when the equivalent functionality is in place. Here's an issue listing what remains to be done to integrate the old pyfact code in to gammapy: #78

Member

cdeil commented Mar 27, 2015

@mapazarr - Thanks!

As you say, a test needs to be added (do this first) and the function needs to be re-written so that it's fast (do this second).

Some suggestions for the test:

  • Use make_empty_image to get an input image.
  • Create some offset and acceptance objects from scratch, or add a utliity function to make it easy to create them for testing (see example here where I've done something similar).

To make it fast, you have to avoid the for loop in Python and instead use numpy arrays.
Use coordinates to get 2D coordinate images and then from that compute a 2D offset image and then evaluate the acceptance curve. For that last part you have to use interpolation, I think, have a look here.

Some info about pyfact.py: it's old spagetti code with useful functionality, but it has to be re-written or re-factored completely and tests / docs added ... this will be a large part of your work ... in most cases it will be simpler if you start from scratch and we just remove pyfact.py at some point when the equivalent functionality is in place. Here's an issue listing what remains to be done to integrate the old pyfact code in to gammapy: #78

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Mar 27, 2015

Member

If you have time to learn more about how to write array-oriented numpy code that avoids for loops in Python, have a look here or here (there's many more resources, but these are pretty good and with excercises I think)

Member

cdeil commented Mar 27, 2015

If you have time to learn more about how to write array-oriented numpy code that avoids for loops in Python, have a look here or here (there's many more resources, but these are pretty good and with excercises I think)

@mapazarr

This comment has been minimized.

Show comment
Hide comment
@mapazarr

mapazarr Mar 29, 2015

Member

About the test. I already had implemented what you say. I uploaded the script that I'm using to an external github repository:
https://github.com/mapazarr/astropy_scripts/blob/master/astropy_scripts/test_fill_acceptance_image.py
My question is: since the test should run automatically, I need to define some "assert", but I don't know the best strategy for it. Should I for instance recompute the interpolation of the acceptance in the test script and check a few bins of the image? (it would use the same implementation as in the method to test, so I'm not sure if the test would be meaningful.)
Another possibility would be to fit the image and check that it delivers a radial function with the same parameters as the one used for the radial acceptance passed as input. This one, with the 2D fit, seems a bit too complicated.

Member

mapazarr commented Mar 29, 2015

About the test. I already had implemented what you say. I uploaded the script that I'm using to an external github repository:
https://github.com/mapazarr/astropy_scripts/blob/master/astropy_scripts/test_fill_acceptance_image.py
My question is: since the test should run automatically, I need to define some "assert", but I don't know the best strategy for it. Should I for instance recompute the interpolation of the acceptance in the test script and check a few bins of the image? (it would use the same implementation as in the method to test, so I'm not sure if the test would be meaningful.)
Another possibility would be to fit the image and check that it delivers a radial function with the same parameters as the one used for the radial acceptance passed as input. This one, with the 2D fit, seems a bit too complicated.

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Mar 29, 2015

Member

@mapazarr – In my experience, writing the asserts for tests of analysis code like most of the code in Gammapy requires roughly as much (sometimes even more) work / thought as implementing the algorithms in the first place.

As you say, duplicating the method implementation in the test doesn't make sense.
Here's a few thoughts / options:

  • Even a test without assert is useful ... if it's in place, travis-ci makes sure that the code runs without raising an exception with various Python / Numpy / Astropy versions.
  • You could assert on the sum of the returned image. Then you have a regression test that makes sure the result is the same when you re-factor the code (e.g. introduce a RadialAcceptanceCurve class that stores the offset and acceptance arrays and has methods to read / write it to a FITS file).
  • In addition to the sum, you could assert on the values of two or three pixels, or on the maximum and minimum pixel position and value.
  • You could use photutils.detection.find_peaks with subpixel=True to get the maximum pixel position in the image with subpixel precision ... it should be very close to the input position and this can catch subtle off-by-one-pixel or off-by-0.5-pixel bugs.
  • You could try to break your function e.g. by passing an image that is larger than the acceptance curve, or a position completely outside your image, and check that your function behaves reasonably.

For now, my recommendation to you would be that you don't implement a ton of tests. This function is most likely not the final API we want (e.g. I think it makes sense to have a RadialAcceptanceCurve class), and you'll have to re-factor your tests if you re-factor your functions. I'm willing to merge this if you have a single assert on the sum of the image, doing more would be nice, but is not required at this point.

One more thing: please try to keep test datasets small. Think how gib the test image / curve needs to be so that you can test everything you want. Personally here I'd have chosen 5 x 5 or 10 x 10 pixel (although probably 3 x 3 would be fine as well), and not 100 x 100 or some big nice image. The advantage is that for small datasets the tests run faster, and if one gets an error, the printed arrays are easier to read / understand.

Member

cdeil commented Mar 29, 2015

@mapazarr – In my experience, writing the asserts for tests of analysis code like most of the code in Gammapy requires roughly as much (sometimes even more) work / thought as implementing the algorithms in the first place.

As you say, duplicating the method implementation in the test doesn't make sense.
Here's a few thoughts / options:

  • Even a test without assert is useful ... if it's in place, travis-ci makes sure that the code runs without raising an exception with various Python / Numpy / Astropy versions.
  • You could assert on the sum of the returned image. Then you have a regression test that makes sure the result is the same when you re-factor the code (e.g. introduce a RadialAcceptanceCurve class that stores the offset and acceptance arrays and has methods to read / write it to a FITS file).
  • In addition to the sum, you could assert on the values of two or three pixels, or on the maximum and minimum pixel position and value.
  • You could use photutils.detection.find_peaks with subpixel=True to get the maximum pixel position in the image with subpixel precision ... it should be very close to the input position and this can catch subtle off-by-one-pixel or off-by-0.5-pixel bugs.
  • You could try to break your function e.g. by passing an image that is larger than the acceptance curve, or a position completely outside your image, and check that your function behaves reasonably.

For now, my recommendation to you would be that you don't implement a ton of tests. This function is most likely not the final API we want (e.g. I think it makes sense to have a RadialAcceptanceCurve class), and you'll have to re-factor your tests if you re-factor your functions. I'm willing to merge this if you have a single assert on the sum of the image, doing more would be nice, but is not required at this point.

One more thing: please try to keep test datasets small. Think how gib the test image / curve needs to be so that you can test everything you want. Personally here I'd have chosen 5 x 5 or 10 x 10 pixel (although probably 3 x 3 would be fine as well), and not 100 x 100 or some big nice image. The advantage is that for small datasets the tests run faster, and if one gets an error, the printed arrays are easier to read / understand.

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Mar 29, 2015

Member

I forgot to mention one very good option to implement tests: check that the output matches some other code. E.g. here you could check against the HESS software. But I'd recommend against it because it takes too much time to set up a small test case in such a way that you can feed it into the HESS software ... which is one reason I started / like Gammapy ... it's simpler to use and more flexible.

Member

cdeil commented Mar 29, 2015

I forgot to mention one very good option to implement tests: check that the output matches some other code. E.g. here you could check against the HESS software. But I'd recommend against it because it takes too much time to set up a small test case in such a way that you can feed it into the HESS software ... which is one reason I started / like Gammapy ... it's simpler to use and more flexible.

@mapazarr

This comment has been minimized.

Show comment
Hide comment
@mapazarr

mapazarr Apr 1, 2015

Member

@cdeil:
I improved the method by avoiding the loop over the pixels. Now it's very fast even for 100x100 pixel images.
The implemented test fails: I committed it for the purpose of asking for assistance. I have a test that tries to compare the integral of the radial acceptance curve (defined as a Gaussian), to the sum of the image, weighted by the pixel areas, which is in practice a discretization of the integral. Im principle, both results should be equivalent (within a certain small epsilon). But in practice, the difference is quite large and I'm not sure where does it come from.
A debug version of the test is posted in:
https://github.com/mapazarr/astropy_scripts/blob/master/astropy_scripts/test_fill_acceptance_image.py
BTW: I am still using 100x100 pixel images for the test because:

  • I need to correctly sample the radial acceptance (Gaussian) curve in order to have precise results to compare the integral and the sum of the image weighted by the pixel areas.
  • With the new method it is fast enough, so it doesn't take too much time to run the test.

Any input is welcome!

Member

mapazarr commented Apr 1, 2015

@cdeil:
I improved the method by avoiding the loop over the pixels. Now it's very fast even for 100x100 pixel images.
The implemented test fails: I committed it for the purpose of asking for assistance. I have a test that tries to compare the integral of the radial acceptance curve (defined as a Gaussian), to the sum of the image, weighted by the pixel areas, which is in practice a discretization of the integral. Im principle, both results should be equivalent (within a certain small epsilon). But in practice, the difference is quite large and I'm not sure where does it come from.
A debug version of the test is posted in:
https://github.com/mapazarr/astropy_scripts/blob/master/astropy_scripts/test_fill_acceptance_image.py
BTW: I am still using 100x100 pixel images for the test because:

  • I need to correctly sample the radial acceptance (Gaussian) curve in order to have precise results to compare the integral and the sum of the image weighted by the pixel areas.
  • With the new method it is fast enough, so it doesn't take too much time to run the test.

Any input is welcome!

@mapazarr

This comment has been minimized.

Show comment
Hide comment
@mapazarr

mapazarr Apr 1, 2015

Member

On another note: could you tell me which is the convention to adopt for the origin in images? Is the origin pixel "1" as for FITS or "0"?
I'll use "0" as for photutils, and most of the code in gammapy (see issue #251).

Member

mapazarr commented Apr 1, 2015

On another note: could you tell me which is the convention to adopt for the origin in images? Is the origin pixel "1" as for FITS or "0"?
I'll use "0" as for photutils, and most of the code in gammapy (see issue #251).

Show outdated Hide outdated gammapy/background/fov.py Outdated
Show outdated Hide outdated gammapy/background/fov.py Outdated
Show outdated Hide outdated gammapy/background/fov.py Outdated
Show outdated Hide outdated gammapy/background/fov.py Outdated
Show outdated Hide outdated gammapy/background/fov.py Outdated
Show outdated Hide outdated gammapy/background/tests/test_fov.py Outdated
Show outdated Hide outdated gammapy/background/tests/test_fov.py Outdated
Show outdated Hide outdated gammapy/background/tests/test_fov.py Outdated
Show outdated Hide outdated gammapy/background/tests/test_fov.py Outdated
Show outdated Hide outdated gammapy/background/tests/test_fov.py Outdated
Show outdated Hide outdated gammapy/background/tests/test_fov.py Outdated
Show outdated Hide outdated gammapy/background/tests/test_fov.py Outdated
Show outdated Hide outdated gammapy/background/tests/test_fov.py Outdated
Show outdated Hide outdated gammapy/background/tests/test_fov.py Outdated
@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Apr 1, 2015

Member

I've pointed out some trivial stuff to improve in inline comments.
About your two core questions:

  1. why is the integral not correct? -> I have to sleep ... I'll try to find time tomorrow morning to have another look.
  2. pixel convention with offset 1 or 0 -> I'd prefer to explain via Skype than by typing lots of text and examples here ... let's Skype briefly sometime next week, email me which day would work for you.
Member

cdeil commented Apr 1, 2015

I've pointed out some trivial stuff to improve in inline comments.
About your two core questions:

  1. why is the integral not correct? -> I have to sleep ... I'll try to find time tomorrow morning to have another look.
  2. pixel convention with offset 1 or 0 -> I'd prefer to explain via Skype than by typing lots of text and examples here ... let's Skype briefly sometime next week, email me which day would work for you.
Show outdated Hide outdated gammapy/background/fov.py Outdated
Show outdated Hide outdated gammapy/background/fov.py Outdated
Show outdated Hide outdated gammapy/background/fov.py Outdated
Show outdated Hide outdated gammapy/background/tests/test_fov.py Outdated

@cdeil cdeil changed the title from Added method for filling image with radial acceptance in FoV bg. to Add function to fill acceptance image from curve Apr 8, 2015

@cdeil cdeil added the feature label Apr 8, 2015

@cdeil cdeil added this to the 0.2 milestone Apr 8, 2015

@mapazarr

This comment has been minimized.

Show comment
Hide comment
@mapazarr

mapazarr Apr 8, 2015

Member

The test is now working. I followed your suggestion and I test on a few values, where the acceptance is defined. I had some trouble figuring out how the binning of the images work but now it's fine.

Member

mapazarr commented Apr 8, 2015

The test is now working. I followed your suggestion and I test on a few values, where the acceptance is defined. I had some trouble figuring out how the binning of the images work but now it's fine.

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Apr 9, 2015

Member

Merging this now. Thanks!

Member

cdeil commented Apr 9, 2015

Merging this now. Thanks!

cdeil added a commit that referenced this pull request Apr 9, 2015

Merge pull request #248 from mapazarr/fill_acc_image
Add function to fill acceptance image from curve

@cdeil cdeil merged commit 5d8447e into gammapy:master Apr 9, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Apr 9, 2015

Member

The fill_acceptance_image function is now listed in the online docs. I hadn't checked the html formatting locally for this PR ... looks good!

Member

cdeil commented Apr 9, 2015

The fill_acceptance_image function is now listed in the online docs. I hadn't checked the html formatting locally for this PR ... looks good!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment