New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add observation sanity check method to DataStore #1082

Merged
merged 4 commits into from Aug 3, 2017

Conversation

Projects
None yet
2 participants
@lmohrmann
Contributor

lmohrmann commented Jul 3, 2017

This simple routine performs some sanity checks on the events and IRF files of the observations in the DataStore.

@cdeil cdeil self-assigned this Jul 3, 2017

@cdeil cdeil added the feature label Jul 3, 2017

@cdeil cdeil added this to the 0.7 milestone Jul 3, 2017

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jul 3, 2017

Member

@lmohrmann - Thanks!

Ideally, I think it would be much better if we always implement checks by putting results in dicts like in
https://github.com/gammapy/gammapy/blob/master/gammapy/irf/psf_check.py
because then those can be logged, but could also be e.g. summarised in a table or more easily used in processing pipelines.

Another suggestion I have is that it would be nice if you add a test that exercises the new code at least once to make sure it's working (and keeps working as we refactor Gammapy). There's this CTA example
https://github.com/gammapy/gammapy-extra/tree/master/test_datasets/cta_1dc
and from HESS we have datasets/hess-crab4-hd-hap-prod2 and datasets/hess-crab4-pa and if none of those works for some reason I could add a more recent one from HAP.

@lmohrmann - Let me know if you want to implement either of these suggestions here, or just merge as-is and leave those suggestions to the future.

Member

cdeil commented Jul 3, 2017

@lmohrmann - Thanks!

Ideally, I think it would be much better if we always implement checks by putting results in dicts like in
https://github.com/gammapy/gammapy/blob/master/gammapy/irf/psf_check.py
because then those can be logged, but could also be e.g. summarised in a table or more easily used in processing pipelines.

Another suggestion I have is that it would be nice if you add a test that exercises the new code at least once to make sure it's working (and keeps working as we refactor Gammapy). There's this CTA example
https://github.com/gammapy/gammapy-extra/tree/master/test_datasets/cta_1dc
and from HESS we have datasets/hess-crab4-hd-hap-prod2 and datasets/hess-crab4-pa and if none of those works for some reason I could add a more recent one from HAP.

@lmohrmann - Let me know if you want to implement either of these suggestions here, or just merge as-is and leave those suggestions to the future.

Show outdated Hide outdated gammapy/data/data_store.py Outdated
Show outdated Hide outdated gammapy/data/data_store.py Outdated
@lmohrmann

This comment has been minimized.

Show comment
Hide comment
@lmohrmann

lmohrmann Jul 4, 2017

Contributor

@cdeil I tried to implement your suggestions, see the new commit. The test in test_data_store.py fails if I use the CTA 1DC data or datasets/hess-crab4-pa because it cannot find the files at the expected locations.

Contributor

lmohrmann commented Jul 4, 2017

@cdeil I tried to implement your suggestions, see the new commit. The test in test_data_store.py fails if I use the CTA 1DC data or datasets/hess-crab4-pa because it cannot find the files at the expected locations.

Show outdated Hide outdated gammapy/data/data_store.py Outdated
Show outdated Hide outdated gammapy/data/data_store.py Outdated
@cdeil

@lmohrmann - sorry for not being responsive. I've left a few inline comments. Let me know if they make sense or if you have any questions.

@lmohrmann

This comment has been minimized.

Show comment
Hide comment
@lmohrmann

lmohrmann Jul 20, 2017

Contributor

I tried to implement your suggestions again, see the latest commit. Let me know if this looks OK now.

Contributor

lmohrmann commented Jul 20, 2017

I tried to implement your suggestions again, see the latest commit. Let me know if this looks OK now.

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Aug 3, 2017

Member

@lmohrmann - Thanks!

Member

cdeil commented Aug 3, 2017

@lmohrmann - Thanks!

@cdeil cdeil merged commit 9e8b671 into gammapy:master Aug 3, 2017

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Aug 3, 2017

Member

@lmohrmann - I see at https://github.com/gammapy/gammapy/pull/1082/commits that you merged master into this feature branch: b6495e4

This is not a problem in this case, we'll leave as-is.

But generally merge commits into feature branches means that overall the git commit history of the project is complex, and makes it harder to review / understand of when which changes happened. So for the future, please rebase on upstream master instead of merging it into your feature branch. We don't have git / developer docs for Gammapy, but it's the same as for Astropy (see http://astropy.readthedocs.io/en/latest/development/workflow/development_workflow.html#rebase-but-only-if-asked) or most other projects on Github.

Member

cdeil commented Aug 3, 2017

@lmohrmann - I see at https://github.com/gammapy/gammapy/pull/1082/commits that you merged master into this feature branch: b6495e4

This is not a problem in this case, we'll leave as-is.

But generally merge commits into feature branches means that overall the git commit history of the project is complex, and makes it harder to review / understand of when which changes happened. So for the future, please rebase on upstream master instead of merging it into your feature branch. We don't have git / developer docs for Gammapy, but it's the same as for Astropy (see http://astropy.readthedocs.io/en/latest/development/workflow/development_workflow.html#rebase-but-only-if-asked) or most other projects on Github.

@cdeil cdeil changed the title from Add routine to DataStore that performs sanity checks on observations to Add observation sanity check method to DataStore Aug 31, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment