New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add detection utilities à la BgStats #569

Merged
merged 13 commits into from Jun 13, 2016

Conversation

Projects
None yet
2 participants
@jjlk
Contributor

jjlk commented Jun 9, 2016

Hi @cdeil @joleroi ,
I added a new class called ObservationStats in order to get statistics from an observation. The test functions are also here.

Julien

@cdeil cdeil added the feature label Jun 9, 2016

@cdeil cdeil added this to the 0.5 milestone Jun 9, 2016

@cdeil cdeil self-assigned this Jun 9, 2016

@cdeil cdeil changed the title from Detection utilities à la BgStats to Add detection utilities à la BgStats Jun 9, 2016

Show outdated Hide outdated gammapy/data/observation_summary.py Outdated
Show outdated Hide outdated gammapy/stats/data.py Outdated
return self.alpha_obs
@property
def sigma(self):

This comment has been minimized.

@cdeil

cdeil Jun 9, 2016

Member

I think that property should be moved to the base class, no?
https://gammapy.readthedocs.io/en/latest/api/gammapy.stats.Stats.html

Note that it will still call the alpha from the subclass ... maybe you can just try moving it to the base class and see if all your tests still pass?

@cdeil

cdeil Jun 9, 2016

Member

I think that property should be moved to the base class, no?
https://gammapy.readthedocs.io/en/latest/api/gammapy.stats.Stats.html

Note that it will still call the alpha from the subclass ... maybe you can just try moving it to the base class and see if all your tests still pass?

This comment has been minimized.

@jjlk

jjlk Jun 12, 2016

Contributor

In the class stats/Stats the property alpha returns the ratio of the exposures (a_on/a_off). When dealing with stacking of multiple observations, we usually average the normalisations with the number of off events, not with the mean acceptance. So I guess we need to re-implement the property alpha in the ObservationStats to deal with the weighting, right ?

@jjlk

jjlk Jun 12, 2016

Contributor

In the class stats/Stats the property alpha returns the ratio of the exposures (a_on/a_off). When dealing with stacking of multiple observations, we usually average the normalisations with the number of off events, not with the mean acceptance. So I guess we need to re-implement the property alpha in the ObservationStats to deal with the weighting, right ?

This comment has been minimized.

@cdeil

cdeil Jun 13, 2016

Member

Ah yes, I forgot about that tricky part for the alpha computation.

Note that at some point I added this standalone function:
http://docs.gammapy.org/en/latest/api/gammapy.stats.combine_stats.html

This has some version of the alpha computation for two observations (probably not debugged / tested).
It should somehow be integrated with the Stats class better.
Maybe it's possible to have a stack classmethod on the base Stats class as well, i.e. follow the design pattern you implemented for ObservationStats?

So I guess we need to re-implement the property alpha in the ObservationStats to deal with the weighting, right ?

I'm not sure. Currently your design is such that the stacked ObservationStats doesn't contain arrays of n_on, a_on, ... but just the stacked values, right? Then the weighting has to happen at stacking time, not alpha computation time, no?

Is it clear to you how to handle / resolve this?
If no, I could have a closer look and think about it, or we delay this issue to a future pull request.
Let me know what you prefer ...

@cdeil

cdeil Jun 13, 2016

Member

Ah yes, I forgot about that tricky part for the alpha computation.

Note that at some point I added this standalone function:
http://docs.gammapy.org/en/latest/api/gammapy.stats.combine_stats.html

This has some version of the alpha computation for two observations (probably not debugged / tested).
It should somehow be integrated with the Stats class better.
Maybe it's possible to have a stack classmethod on the base Stats class as well, i.e. follow the design pattern you implemented for ObservationStats?

So I guess we need to re-implement the property alpha in the ObservationStats to deal with the weighting, right ?

I'm not sure. Currently your design is such that the stacked ObservationStats doesn't contain arrays of n_on, a_on, ... but just the stacked values, right? Then the weighting has to happen at stacking time, not alpha computation time, no?

Is it clear to you how to handle / resolve this?
If no, I could have a closer look and think about it, or we delay this issue to a future pull request.
Let me know what you prefer ...

Show outdated Hide outdated gammapy/data/observation_stats.py Outdated
Show outdated Hide outdated gammapy/data/observation_stats.py Outdated
Show outdated Hide outdated gammapy/data/observation_stats.py Outdated
@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jun 9, 2016

Member

I've left a bunch of minor comments on code cleanup.

I'll try to find some time tomorrow morning to think about the API and read through the implementation and tests in detail.

Member

cdeil commented Jun 9, 2016

I've left a bunch of minor comments on code cleanup.

I'll try to find some time tomorrow morning to think about the API and read through the implementation and tests in detail.

jjlk added some commits Jun 12, 2016

@jjlk

This comment has been minimized.

Show comment
Hide comment
@jjlk

jjlk Jun 12, 2016

Contributor

Hi @cdeil ,
I took into account your comments. I also implemented the class ObservationSummary and as I said above:

I tried to implement this class and keep it simple. Now it take an ObservationStatsList and do some standard plots of detection. Maybe we could wrap it in an other class taking some options or a configuration file, to deal with the regions and the background substraction, as you suggested if I remember correctly. We could also add the ObservationTableSummary utilities.

Maybe we could find some time do discuss this tomorrow via slack or Skype ?

Cheers !

Contributor

jjlk commented Jun 12, 2016

Hi @cdeil ,
I took into account your comments. I also implemented the class ObservationSummary and as I said above:

I tried to implement this class and keep it simple. Now it take an ObservationStatsList and do some standard plots of detection. Maybe we could wrap it in an other class taking some options or a configuration file, to deal with the regions and the background substraction, as you suggested if I remember correctly. We could also add the ObservationTableSummary utilities.

Maybe we could find some time do discuss this tomorrow via slack or Skype ?

Cheers !

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jun 13, 2016

Member

There's two test fails on travis-ci:
https://travis-ci.org/gammapy/gammapy/jobs/137101665#L944
You should be able to reproduce those locally by running:

python setup.py test -V -t gammapy/data/tests/test_observation_stats.py

Fixing those is the most important part so that this pull request can go in.
Other things (stacked alpha computation, observation stats list to / from table, observation summary, ...) can be done in future pull requests one addition at a time.

Member

cdeil commented Jun 13, 2016

There's two test fails on travis-ci:
https://travis-ci.org/gammapy/gammapy/jobs/137101665#L944
You should be able to reproduce those locally by running:

python setup.py test -V -t gammapy/data/tests/test_observation_stats.py

Fixing those is the most important part so that this pull request can go in.
Other things (stacked alpha computation, observation stats list to / from table, observation summary, ...) can be done in future pull requests one addition at a time.

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jun 13, 2016

Member

I see that you've implemented one version of ObservationStatsList to arrays in the ObservationSummary __init__ and _init_values methods.

I'm not sure if that is the best place for this code ... naively I would have expected methods ObservationStatsList.to_table and ObservationStatsList.from_table with an option on whether it's cumulative or not. But overall I'm not clear on where the distinction between ObservationStatsList and ObservationStats lies. I guess it's up to you @jjlk .
Eventually we also want a docs/data/stats.rst tutorial page that shows an example of how to use these classes.

But this PR is already 500 lines and hard for me to review, so I'm happy to merge as soon as tests pass and you think it's ready @jjlk , and then I'll use it for a bit (e.g. for the data release scripts) and then can give better feedback.

Thank you for this!

Member

cdeil commented Jun 13, 2016

I see that you've implemented one version of ObservationStatsList to arrays in the ObservationSummary __init__ and _init_values methods.

I'm not sure if that is the best place for this code ... naively I would have expected methods ObservationStatsList.to_table and ObservationStatsList.from_table with an option on whether it's cumulative or not. But overall I'm not clear on where the distinction between ObservationStatsList and ObservationStats lies. I guess it's up to you @jjlk .
Eventually we also want a docs/data/stats.rst tutorial page that shows an example of how to use these classes.

But this PR is already 500 lines and hard for me to review, so I'm happy to merge as soon as tests pass and you think it's ready @jjlk , and then I'll use it for a bit (e.g. for the data release scripts) and then can give better feedback.

Thank you for this!

@jjlk

This comment has been minimized.

Show comment
Hide comment
@jjlk

jjlk Jun 13, 2016

Contributor

Hi @cdeil , I fixed the ObservationStats::str class and the test is fine now. I'll tackle your other comments in new pull requests

Contributor

jjlk commented Jun 13, 2016

Hi @cdeil , I fixed the ObservationStats::str class and the test is fine now. I'll tackle your other comments in new pull requests

Show outdated Hide outdated gammapy/data/observation_summary.py Outdated
Show outdated Hide outdated gammapy/data/observation_stats.py Outdated
@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jun 13, 2016

Member

I promise I'll stop suggesting thing now on this PR and we'll merge as soon as travis-ci passes.
Thanks for your patience, @jjlk !

Member

cdeil commented Jun 13, 2016

I promise I'll stop suggesting thing now on this PR and we'll merge as soon as travis-ci passes.
Thanks for your patience, @jjlk !

@jjlk

This comment has been minimized.

Show comment
Hide comment
@jjlk

jjlk Jun 13, 2016

Contributor

Haha !
@cdeil : changes done

Contributor

jjlk commented Jun 13, 2016

Haha !
@cdeil : changes done

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jun 13, 2016

Member

Looks good. I'm merging this in. Thanks!

Member

cdeil commented Jun 13, 2016

Looks good. I'm merging this in. Thanks!

@cdeil cdeil merged commit e1b31ad into gammapy:master Jun 13, 2016

0 of 2 checks passed

continuous-integration/appveyor/pr Waiting for AppVeyor build to complete
Details
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jun 13, 2016

Member

@jjlk - I've done some minor cleanup in master (commits 25fe29d, 96e3cef, 110a741) for things I noticed when opening up the files locally in PyCharm. So please make sure to re-start from latest master if you continue editing these classes.

One more thing that PyCharm pointed out to me when I browsed your code locally is that the Parameters description for gammapy.data.ObservationStats.from_target is incorrect.
The method takes obs, target, and bg_estimate, and the parameter description says obs_table, target_pos and bg_method.
I didn't check what is actually the case or fix that one. @jjlk - Maybe you can fix this in your next PR. :-)

Member

cdeil commented Jun 13, 2016

@jjlk - I've done some minor cleanup in master (commits 25fe29d, 96e3cef, 110a741) for things I noticed when opening up the files locally in PyCharm. So please make sure to re-start from latest master if you continue editing these classes.

One more thing that PyCharm pointed out to me when I browsed your code locally is that the Parameters description for gammapy.data.ObservationStats.from_target is incorrect.
The method takes obs, target, and bg_estimate, and the parameter description says obs_table, target_pos and bg_method.
I didn't check what is actually the case or fix that one. @jjlk - Maybe you can fix this in your next PR. :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment