New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve spectrum analysis #403

Merged
merged 22 commits into from Jan 15, 2016

Conversation

Projects
None yet
2 participants
@joleroi
Contributor

joleroi commented Dec 14, 2015

This PR introduces a new background method to the spectral fitting utilities: OFF vectors from Background models.

The appropriate statistics in Sherpa (WStat vs. cash) is not yet available. Default stat is set to CASH

  • add method to compute OFF vec from background model
  • add debug plots
  • add functionality to exclude runs without sufficient number of reflected regions
  • add copy function to SpectrumAnalysis (to try different bkg methods)

@joleroi joleroi self-assigned this Dec 14, 2015

@joleroi joleroi added the feature label Dec 14, 2015

@joleroi joleroi added this to the 0.4 milestone Dec 14, 2015

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi Dec 18, 2015

Contributor

I would like to merge this today. Comments welcome

Contributor

joleroi commented Dec 18, 2015

I would like to merge this today. Comments welcome

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi Dec 18, 2015

Contributor

@cdeil : one question - in gammapy-extra/datasets/hess-crab4_pha I have some example PHA data. In order to use this for tests I set the reference to the RMF and ARF files in the PHA header relative to $GAMMAPY_EXTRA.
Now I get this error
https://travis-ci.org/gammapy/gammapy/jobs/97637674#L1431

Can I in principle use gammapy-extra this way or does it only work via gammapy.datasets.gammapy-extra. If the latter is the case, I don't know how to fix this test

Contributor

joleroi commented Dec 18, 2015

@cdeil : one question - in gammapy-extra/datasets/hess-crab4_pha I have some example PHA data. In order to use this for tests I set the reference to the RMF and ARF files in the PHA header relative to $GAMMAPY_EXTRA.
Now I get this error
https://travis-ci.org/gammapy/gammapy/jobs/97637674#L1431

Can I in principle use gammapy-extra this way or does it only work via gammapy.datasets.gammapy-extra. If the latter is the case, I don't know how to fix this test

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi Dec 18, 2015

Contributor

@cdeil: I found the error, it was a bug on my side

Contributor

joleroi commented Dec 18, 2015

@cdeil: I found the error, it was a bug on my side

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jan 7, 2016

Member

@joleroi - Rebase on master, please.

Member

cdeil commented Jan 7, 2016

@joleroi - Rebase on master, please.

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi Jan 7, 2016

Contributor

Tests still fail because of #408 . I guess that used to happen only locally, but I guess now it's also a problem for travis (which is a good thing I guess). I have no idea how to attack this, though.

Contributor

joleroi commented Jan 7, 2016

Tests still fail because of #408 . I guess that used to happen only locally, but I guess now it's also a problem for travis (which is a good thing I guess). I have no idea how to attack this, though.

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi Jan 8, 2016

Contributor

Any comments before I merge?

Contributor

joleroi commented Jan 8, 2016

Any comments before I merge?

Show outdated Hide outdated gammapy/region/circle.py Outdated
Show outdated Hide outdated gammapy/region/tests/test_circle.py Outdated
Show outdated Hide outdated gammapy/region/tests/test_circle.py Outdated
Show outdated Hide outdated gammapy/spectrum/spectrum_analysis.py Outdated
Show outdated Hide outdated gammapy/spectrum/spectrum_analysis.py Outdated
Show outdated Hide outdated gammapy/spectrum/spectrum_analysis.py Outdated
Show outdated Hide outdated gammapy/spectrum/spectrum_analysis.py Outdated
self.energy_dispersion.write(str(outdir/rmffile), energy_unit='keV',
clobber=clobber)
def plot_exclusion_mask(self, size=None, **kwargs):

This comment has been minimized.

@cdeil

cdeil Jan 8, 2016

Member

Pass ax so that it's possible to easily create this plot as one of several panels in a larger Figure?

@cdeil

cdeil Jan 8, 2016

Member

Pass ax so that it's possible to easily create this plot as one of several panels in a larger Figure?

This comment has been minimized.

@joleroi

joleroi Jan 14, 2016

Contributor

I had a discussion about this with Axel, the problem is that I need a WCS axis object, right?
This object has to know the transformation defined in the Exclusion mask header apparently due to some binning issue. So, how can I pass an axis and then give it the WCS info from the exclusion mask?

@joleroi

joleroi Jan 14, 2016

Contributor

I had a discussion about this with Axel, the problem is that I need a WCS axis object, right?
This object has to know the transformation defined in the Exclusion mask header apparently due to some binning issue. So, how can I pass an axis and then give it the WCS info from the exclusion mask?

This comment has been minimized.

@cdeil

cdeil Jan 14, 2016

Member

I vaguely remember discussing this in the past as well and don't know if we found a good solution.
Maybe split this out in a separate issue and leave as-is for now?

@cdeil

cdeil Jan 14, 2016

Member

I vaguely remember discussing this in the past as well and don't know if we found a good solution.
Maybe split this out in a separate issue and leave as-is for now?

Show outdated Hide outdated gammapy/spectrum/spectrum_analysis.py Outdated
self._model = model
@property
def statistic(self):

This comment has been minimized.

@cdeil

cdeil Jan 8, 2016

Member

In __init__ you called the data member statistic.
Now you're creating a property statistic that accesses a data member _statistic.
Is this a bug?

@cdeil

cdeil Jan 8, 2016

Member

In __init__ you called the data member statistic.
Now you're creating a property statistic that accesses a data member _statistic.
Is this a bug?

This comment has been minimized.

@joleroi

joleroi Jan 14, 2016

Contributor

Yes

@joleroi

joleroi Jan 14, 2016

Contributor

Yes

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jan 8, 2016

Member

I've left a few inline comments.

Maybe change the PR title from "OFF vector from cube background model" to "Improve spectrum analysis" since this PR now does several important improvements?

Also this comment in the PR description is cryptic:

The appropriate statistics in Sherpa (WStat vs. cash) is not yet available. Default stat is set to CASH

When using a background spectral template extracted from stacked off runs, using CASH is the right thing to do, and it's available, and you're using it. Maybe rephrase?

If you have time, please look over the coverage of the files you've worked on:
https://coveralls.io/builds/4663382
Maybe this could be improved a bit?
https://coveralls.io/builds/4663382/source?filename=gammapy%2Fscripts%2Fspectrum_pipe.py

Wouldn't it be better to split the SpectrumAnalysis.make_comparison_plot functionality out into a standalone function or class? I.e. have SpectrumAnalysis only be responsible for running one analysis, have a way to serialise the results, and then have separate functions / classes to compare analyses?
Of course it's OK to do this in future PRs ... I just saw that this here for the first time:
https://coveralls.io/builds/4663382/source?filename=gammapy%2Fscripts%2Fspectrum_pipe.py#L139

Member

cdeil commented Jan 8, 2016

I've left a few inline comments.

Maybe change the PR title from "OFF vector from cube background model" to "Improve spectrum analysis" since this PR now does several important improvements?

Also this comment in the PR description is cryptic:

The appropriate statistics in Sherpa (WStat vs. cash) is not yet available. Default stat is set to CASH

When using a background spectral template extracted from stacked off runs, using CASH is the right thing to do, and it's available, and you're using it. Maybe rephrase?

If you have time, please look over the coverage of the files you've worked on:
https://coveralls.io/builds/4663382
Maybe this could be improved a bit?
https://coveralls.io/builds/4663382/source?filename=gammapy%2Fscripts%2Fspectrum_pipe.py

Wouldn't it be better to split the SpectrumAnalysis.make_comparison_plot functionality out into a standalone function or class? I.e. have SpectrumAnalysis only be responsible for running one analysis, have a way to serialise the results, and then have separate functions / classes to compare analyses?
Of course it's OK to do this in future PRs ... I just saw that this here for the first time:
https://coveralls.io/builds/4663382/source?filename=gammapy%2Fscripts%2Fspectrum_pipe.py#L139

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi Jan 11, 2016

Contributor

Thank for the comments, I will go trough them today. About your general comments:

  • Concerning the statistics issue: At the moment ONLY cash is availabe for all spectral fits. The Version of sherpa I have installed (4.7) does not have WStat, and I don't manage to install the head version
checking for flex... flex
checking lex output file root... configure: error: cannot find output from flex; giving up
configure: error: ./configure failed for region-4.7

This looks related https://gist.github.com/cdeil/39e9404e91954cb67270. Suggestions?

  • Continued ...

When using a background spectral template extracted from stacked off runs, using CASH is the right thing to do

Yes and I don't know, the background model gives non-integer couns, sherpa only understands PHA (i.e. integer counts). I am happy for any suggestions what to do 😁

  • Make comparison plot

Wouldn't it be better to split the SpectrumAnalysis.make_comparison_plot functionality out into a standalone function or class?

Of course this is just something I hacked together. It will go away soon!

Contributor

joleroi commented Jan 11, 2016

Thank for the comments, I will go trough them today. About your general comments:

  • Concerning the statistics issue: At the moment ONLY cash is availabe for all spectral fits. The Version of sherpa I have installed (4.7) does not have WStat, and I don't manage to install the head version
checking for flex... flex
checking lex output file root... configure: error: cannot find output from flex; giving up
configure: error: ./configure failed for region-4.7

This looks related https://gist.github.com/cdeil/39e9404e91954cb67270. Suggestions?

  • Continued ...

When using a background spectral template extracted from stacked off runs, using CASH is the right thing to do

Yes and I don't know, the background model gives non-integer couns, sherpa only understands PHA (i.e. integer counts). I am happy for any suggestions what to do 😁

  • Make comparison plot

Wouldn't it be better to split the SpectrumAnalysis.make_comparison_plot functionality out into a standalone function or class?

Of course this is just something I hacked together. It will go away soon!

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jan 11, 2016

Member

@joleroi - Could you please make EDISP2D spec a priority because of the upcoming telcon (open-gamma-ray-astro/gamma-astro-data-formats#16 (comment))?

Concerning the Sherpa build issue and how to set the non-integer expected counts background model in Sherpa ... I don't know, but I could come by later today and we try to figure it out (or maybe also Peter or Regis can help with the spectrum background model issue).

Member

cdeil commented Jan 11, 2016

@joleroi - Could you please make EDISP2D spec a priority because of the upcoming telcon (open-gamma-ray-astro/gamma-astro-data-formats#16 (comment))?

Concerning the Sherpa build issue and how to set the non-integer expected counts background model in Sherpa ... I don't know, but I could come by later today and we try to figure it out (or maybe also Peter or Regis can help with the spectrum background model issue).

@joleroi joleroi changed the title from OFF vector from cube background model to Improve spectrum analysis Jan 14, 2016

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi Jan 14, 2016

Contributor

@cdeil Do you know the the tests behave so strangely?

Contributor

joleroi commented Jan 14, 2016

@cdeil Do you know the the tests behave so strangely?

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jan 14, 2016

Member

Do you mean on the CI servers?

You have to rebase this branch:

screen shot 2016-01-14 at 09 46 00

If you mean local testing, I could come by later today and help debug it.

Member

cdeil commented Jan 14, 2016

Do you mean on the CI servers?

You have to rebase this branch:

screen shot 2016-01-14 at 09 46 00

If you mean local testing, I could come by later today and help debug it.

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi Jan 14, 2016

Contributor

I meant on the CI servers. Was it always the case that the need for rebase made the test not run? But anyhow, thanks!

Contributor

joleroi commented Jan 14, 2016

I meant on the CI servers. Was it always the case that the need for rebase made the test not run? But anyhow, thanks!

@joleroi

This comment has been minimized.

Show comment
Hide comment
@joleroi

joleroi Jan 14, 2016

Contributor

Can I merge?

Contributor

joleroi commented Jan 14, 2016

Can I merge?

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jan 14, 2016

Member

👍 to merge!

(I didn't review this in detail or try it out because I don't have time at the moment. So I might have comments in the future.)

Member

cdeil commented Jan 14, 2016

👍 to merge!

(I didn't review this in detail or try it out because I don't have time at the moment. So I might have comments in the future.)

joleroi added a commit that referenced this pull request Jan 15, 2016

@joleroi joleroi merged commit da10a84 into gammapy:master Jan 15, 2016

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@joleroi joleroi deleted the joleroi:off_from_background branch Jan 15, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment