New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document observation tables and improve gammapy.obs #278

Merged
merged 15 commits into from Jun 12, 2015

Conversation

Projects
None yet
3 participants
@mapazarr
Member

mapazarr commented Jun 2, 2015

The method still needs some work and clean-up. I am committing a preliminary version in order to trigger a discussion in the hangout tomorrow. Please don't look/comment on the minor details yet, but rather the API and the way of how to fill the obs table. I think maybe some other methods should be defined, like FillOneObservation and separate the building of a obs list from the faking of a list.
The method has a working test.

Show outdated Hide outdated gammapy/datasets/make.py Outdated
Show outdated Hide outdated gammapy/datasets/make.py Outdated
Show outdated Hide outdated gammapy/datasets/make.py Outdated
Show outdated Hide outdated gammapy/datasets/make.py Outdated
Show outdated Hide outdated gammapy/datasets/make.py Outdated
Show outdated Hide outdated gammapy/datasets/make.py Outdated
@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jun 3, 2015

Member

While you're working on this, could you pleas also update this observation table format spec / description?
https://gammapy.readthedocs.org/en/latest/dataformats/observation_lists.html

Don't worry too much about what's in runinfo.fits now ... the important thing is to come up with a good, documented format. Once that is written down we can adapt the existing file we have to this format.

Member

cdeil commented Jun 3, 2015

While you're working on this, could you pleas also update this observation table format spec / description?
https://gammapy.readthedocs.org/en/latest/dataformats/observation_lists.html

Don't worry too much about what's in runinfo.fits now ... the important thing is to come up with a good, documented format. Once that is written down we can adapt the existing file we have to this format.

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jun 3, 2015

Member

It would be nice if you could fake TELESCOPE_PATTERN (or at least N_TEL) and MUON_EFFICIENCY as well, these are probably parameters we want to use to group observations soon.

Otherwise 👍 , this is heading in the direction I had in mind when we talked about this.

Member

cdeil commented Jun 3, 2015

It would be nice if you could fake TELESCOPE_PATTERN (or at least N_TEL) and MUON_EFFICIENCY as well, these are probably parameters we want to use to group observations soon.

Otherwise 👍 , this is heading in the direction I had in mind when we talked about this.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jun 4, 2015

Coverage Status

Coverage increased (+0.19%) to 43.32% when pulling 541c57c on mapazarr:gen_obs_tables into 0883871 on gammapy:master.

coveralls commented Jun 4, 2015

Coverage Status

Coverage increased (+0.19%) to 43.32% when pulling 541c57c on mapazarr:gen_obs_tables into 0883871 on gammapy:master.

@mapazarr

This comment has been minimized.

Show comment
Hide comment
@mapazarr

mapazarr Jun 4, 2015

Member

I fixed the error on travis-ci. I will continue working on this tomorrow.

Member

mapazarr commented Jun 4, 2015

I fixed the error on travis-ci. I will continue working on this tomorrow.

@mapazarr

This comment has been minimized.

Show comment
Hide comment
@mapazarr

mapazarr Jun 4, 2015

Member

@cdeil @mimayer
Do we need TSTOP and ONTIME? We can calculate ONTIME as TSTOP - TSTART, right? Or TSTOP as TSTART + ONTIME.

Member

mapazarr commented Jun 4, 2015

@cdeil @mimayer
Do we need TSTOP and ONTIME? We can calculate ONTIME as TSTOP - TSTART, right? Or TSTOP as TSTART + ONTIME.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jun 4, 2015

Coverage Status

Coverage increased (+0.34%) to 43.46% when pulling 895de1e on mapazarr:gen_obs_tables into 0883871 on gammapy:master.

coveralls commented Jun 4, 2015

Coverage Status

Coverage increased (+0.34%) to 43.46% when pulling 895de1e on mapazarr:gen_obs_tables into 0883871 on gammapy:master.

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jun 4, 2015

Member

I'd say make TSTART required. You want to know the run date e.g. for time selection in findruns "give me all runs from 2005 to 2007".
TSTOP and ONTIME are sometimes of interest, so should be optional. You can still define them in this format, so that everyone will use the same names if they do add those columns, and not one guy calls it TSTOP and then another gal calls it TEND.

Member

cdeil commented Jun 4, 2015

I'd say make TSTART required. You want to know the run date e.g. for time selection in findruns "give me all runs from 2005 to 2007".
TSTOP and ONTIME are sometimes of interest, so should be optional. You can still define them in this format, so that everyone will use the same names if they do add those columns, and not one guy calls it TSTOP and then another gal calls it TEND.

@cdeil cdeil added the feature label Jun 4, 2015

@cdeil cdeil added this to the 0.3 milestone Jun 4, 2015

Show outdated Hide outdated gammapy/utils/tests/test_time.py Outdated
@@ -0,0 +1,21 @@
# Licensed under a 3-clause BSD style license - see LICENSE.rst
from __future__ import (absolute_import, division, print_function,
unicode_literals)

This comment has been minimized.

@mapazarr

mapazarr Jun 4, 2015

Member

This comment has been minimized.

@cdeil

cdeil Jun 4, 2015

Member

Yes, please add this to any new file.

It's an Astropy coding guideline, and it also makes sense: if you don't add it, it is possible to get slightly different behaviour on Python 2 and 3 (e.g. 1/2 ==0 on Python 2 or 1/2 == 0.5 on Python 3). If test coverage is not very high and you don't put in the right test dataset, issues can go unnoticed.

I know it's annoying boilerplate code, but thinking about / discussing where you need or might need it in the future is worse than just pasting it into every file.

@cdeil

cdeil Jun 4, 2015

Member

Yes, please add this to any new file.

It's an Astropy coding guideline, and it also makes sense: if you don't add it, it is possible to get slightly different behaviour on Python 2 and 3 (e.g. 1/2 ==0 on Python 2 or 1/2 == 0.5 on Python 3). If test coverage is not very high and you don't put in the right test dataset, issues can go unnoticed.

I know it's annoying boilerplate code, but thinking about / discussing where you need or might need it in the future is worse than just pasting it into every file.

This comment has been minimized.

@cdeil

cdeil Jun 4, 2015

Member

We can remove it when we drop Python 2 support. I.e. when Sherpa supports Python 3 and > 90% of our users are on Python 3, i.e. maybe in a few years, maybe never. :-)

@cdeil

cdeil Jun 4, 2015

Member

We can remove it when we drop Python 2 support. I.e. when Sherpa supports Python 3 and > 90% of our users are on Python 3, i.e. maybe in a few years, maybe never. :-)

This comment has been minimized.

@mapazarr

mapazarr Jun 4, 2015

Member

Ok.

@mapazarr

mapazarr Jun 4, 2015

Member

Ok.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jun 4, 2015

Coverage Status

Coverage increased (+0.36%) to 43.48% when pulling bee18cc on mapazarr:gen_obs_tables into 0883871 on gammapy:master.

coveralls commented Jun 4, 2015

Coverage Status

Coverage increased (+0.36%) to 43.48% when pulling bee18cc on mapazarr:gen_obs_tables into 0883871 on gammapy:master.

@mapazarr

This comment has been minimized.

Show comment
Hide comment
@mapazarr

mapazarr Jun 5, 2015

Member

@cdeil I started working on the doc about the format for obs lists. Please have a look at the "TODOs" in the code.

Member

mapazarr commented Jun 5, 2015

@cdeil I started working on the doc about the format for obs lists. Please have a look at the "TODOs" in the code.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Jun 6, 2015

Coverage Status

Coverage increased (+0.35%) to 43.48% when pulling cd55c27 on mapazarr:gen_obs_tables into 0883871 on gammapy:master.

coveralls commented Jun 6, 2015

Coverage Status

Coverage increased (+0.35%) to 43.48% when pulling cd55c27 on mapazarr:gen_obs_tables into 0883871 on gammapy:master.

Show outdated Hide outdated docs/dataformats/observation_lists.rst Outdated
Show outdated Hide outdated docs/dataformats/observation_lists.rst Outdated
Show outdated Hide outdated docs/dataformats/observation_lists.rst Outdated
Show outdated Hide outdated docs/dataformats/observation_lists.rst Outdated
Show outdated Hide outdated docs/dataformats/observation_lists.rst Outdated
Show outdated Hide outdated docs/dataformats/observation_lists.rst Outdated
Show outdated Hide outdated docs/dataformats/observation_lists.rst Outdated
Show outdated Hide outdated docs/dataformats/observation_lists.rst Outdated
Show outdated Hide outdated docs/dataformats/observation_lists.rst Outdated
Show outdated Hide outdated gammapy/datasets/make.py Outdated
Show outdated Hide outdated gammapy/datasets/make.py Outdated
Show outdated Hide outdated gammapy/datasets/make.py Outdated
Show outdated Hide outdated gammapy/datasets/tests/test_make.py Outdated
Show outdated Hide outdated gammapy/obs/observation.py Outdated
* ...
convenience methods. The format of the observation table
is described in :ref:`dataformats_observation_lists`.

This comment has been minimized.

@cdeil

cdeil Jun 11, 2015

Member

This is actually not obvious to me: does this link work in the HTML docs?
(I don't have time to build the docs locally and check myself.)

@cdeil

cdeil Jun 11, 2015

Member

This is actually not obvious to me: does this link work in the HTML docs?
(I don't have time to build the docs locally and check myself.)

This comment has been minimized.

@mapazarr

mapazarr Jun 11, 2015

Member

Yes it works (I had already checked). Nice, hu? :-)

@mapazarr

mapazarr Jun 11, 2015

Member

Yes it works (I had already checked). Nice, hu? :-)

This comment has been minimized.

@cdeil

cdeil Jun 12, 2015

Member

Yes, this is nice. One can even cross-link to Astropy or other doc sets via intersphinx.
I didn't know this a year ago, so there's plenty of places in Gammapy where one could cross-link high-level and API docs. Please do when you see opportunities for improvement or centralising duplicated documentation.

@cdeil

cdeil Jun 12, 2015

Member

Yes, this is nice. One can even cross-link to Astropy or other doc sets via intersphinx.
I didn't know this a year ago, so there's plenty of places in Gammapy where one could cross-link high-level and API docs. Please do when you see opportunities for improvement or centralising duplicated documentation.

Show outdated Hide outdated gammapy/obs/observation.py Outdated
Show outdated Hide outdated gammapy/utils/tests/test_time.py Outdated
Show outdated Hide outdated gammapy/utils/tests/test_time.py Outdated
@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jun 11, 2015

Member

@mapazarr – I did one more review of this PR and left inline comments. Nothing major, you should be able to finalise this PR today or tomorrow.
If you're not sure what exactly to write about times in Gammapy, just write two or three sentences ("should use Time class everywhere, when storing times in files use UTC or MET with a reference time") now and then we'll update this in the coming months when the relevant CTA spec is written by Catherine Boisson and me.

Member

cdeil commented Jun 11, 2015

@mapazarr – I did one more review of this PR and left inline comments. Nothing major, you should be able to finalise this PR today or tomorrow.
If you're not sure what exactly to write about times in Gammapy, just write two or three sentences ("should use Time class everywhere, when storing times in files use UTC or MET with a reference time") now and then we'll update this in the coming months when the relevant CTA spec is written by Catherine Boisson and me.

Added a few more tests to the dummy obs table generator. Added doc ab…
…out times in Gammapy and a glossary. Fixed many small formatting isues.
@mapazarr

This comment has been minimized.

Show comment
Hide comment
@mapazarr

mapazarr Jun 11, 2015

Member

OK, I implemented all issues. @cdeil: I hope you can now merge the PR.
I also figured out how to put the doc in google drive. The link to the doc is here:
www.googledrive.com/host/0BzXHZQx0oCLBfmNwR294aFk5R0NuWlhTaDl5Q2Y0ajFJMGFXX0YxeW1CRlhjQ1FfelhFdEE/
It still is very slow during the upload, not because of the size of the docs dir (14 MB), but because of the amount of files (~1300). @cdeil @adonath: do you know if there is a chance to upload compressed to google drive dirs and expand them once they are uploaded?
Ok, I tried https://cloudconvert.com/ but it is still very slow because it has to upload the files into drive (it doesn't act inside it).
The idea with the tarball is that we split the work, so:

  • You don't have to install the branch and build the doc, just download and open the file.
  • I don't have to wait so long for the upload (I hope it doesn't break).

In any case, please let mo know if youcan see the contents of the link I gave you.

Member

mapazarr commented Jun 11, 2015

OK, I implemented all issues. @cdeil: I hope you can now merge the PR.
I also figured out how to put the doc in google drive. The link to the doc is here:
www.googledrive.com/host/0BzXHZQx0oCLBfmNwR294aFk5R0NuWlhTaDl5Q2Y0ajFJMGFXX0YxeW1CRlhjQ1FfelhFdEE/
It still is very slow during the upload, not because of the size of the docs dir (14 MB), but because of the amount of files (~1300). @cdeil @adonath: do you know if there is a chance to upload compressed to google drive dirs and expand them once they are uploaded?
Ok, I tried https://cloudconvert.com/ but it is still very slow because it has to upload the files into drive (it doesn't act inside it).
The idea with the tarball is that we split the work, so:

  • You don't have to install the branch and build the doc, just download and open the file.
  • I don't have to wait so long for the upload (I hope it doesn't break).

In any case, please let mo know if youcan see the contents of the link I gave you.

Show outdated Hide outdated docs/dataformats/index.rst Outdated
Show outdated Hide outdated gammapy/utils/time.py Outdated
muon_efficiency_min = 0.6
muon_efficiency_max = 1.0
muon_efficiency = np.random.random(
len(obs_id)) * (muon_efficiency_max - muon_efficiency_min) + muon_efficiency_min

This comment has been minimized.

@cdeil

cdeil Jun 11, 2015

Member

I don't like multi-line expressions. They are hard to read. With this line break especially.
Please use a temp variable instead to avoid the line break.

@cdeil

cdeil Jun 11, 2015

Member

I don't like multi-line expressions. They are hard to read. With this line break especially.
Please use a temp variable instead to avoid the line break.

This comment has been minimized.

@mapazarr

mapazarr Jun 11, 2015

Member

I also don't, but I also don't like breaking a simple expression into many lines of code with temporary variables: it makes the code very hard to read. This was the suggestion of autopep8 on long lines of code. I'd rather have it like this instead of with many intermediate steps.

@mapazarr

mapazarr Jun 11, 2015

Member

I also don't, but I also don't like breaking a simple expression into many lines of code with temporary variables: it makes the code very hard to read. This was the suggestion of autopep8 on long lines of code. I'd rather have it like this instead of with many intermediate steps.

This comment has been minimized.

@cdeil

cdeil Jun 12, 2015

Member

Coding style and what is more or less readable code is always a matter of taste.
For now I'm calling the shots in Gammapy and am defining the coding style.
I'll clean this up in #285.

Note that this is not unusual, as developers join and leave to work on projects they have to adapt to the existing coding style or the one maintainers aim for even if they don't like it, because consistency is important.

😄

@cdeil

cdeil Jun 12, 2015

Member

Coding style and what is more or less readable code is always a matter of taste.
For now I'm calling the shots in Gammapy and am defining the coding style.
I'll clean this up in #285.

Note that this is not unusual, as developers join and leave to work on projects they have to adapt to the existing coding style or the one maintainers aim for even if they don't like it, because consistency is important.

😄

obs_table['RA'] = sky_coord.ra
obs_table['DEC'] = sky_coord.dec
# positions

This comment has been minimized.

@cdeil

cdeil Jun 11, 2015

Member

What's up with this comment? Remove?

@cdeil

cdeil Jun 11, 2015

Member

What's up with this comment? Remove?

This comment has been minimized.

@mapazarr

mapazarr Jun 12, 2015

Member

Fixed.

@mapazarr

mapazarr Jun 12, 2015

Member

Fixed.

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jun 11, 2015

Member

A few more very minor comments inline. I'll merge this tomorrow morning after travis-ci passes.

Yes, docs have lots of files and upload takes a few minutes. I think that's acceptable for large PRs where having the docs online helps.
As I said, I don't want to download the tarball ... if I have to get the docs locally I prefer checking out your branch via git and running python setup.py build_sphinx.

😴

Member

cdeil commented Jun 11, 2015

A few more very minor comments inline. I'll merge this tomorrow morning after travis-ci passes.

Yes, docs have lots of files and upload takes a few minutes. I think that's acceptable for large PRs where having the docs online helps.
As I said, I don't want to download the tarball ... if I have to get the docs locally I prefer checking out your branch via git and running python setup.py build_sphinx.

😴

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jun 11, 2015

Member

One more thing ... I see you've edited the comment above.
Github doesn't send email notifications for comment edits, so usually I won't see any extra things you write. It's better to make a new comment if you forgot to say something so that I'll see it.

Member

cdeil commented Jun 11, 2015

One more thing ... I see you've edited the comment above.
Github doesn't send email notifications for comment edits, so usually I won't see any extra things you write. It's better to make a new comment if you forgot to say something so that I'll see it.

@mapazarr

This comment has been minimized.

Show comment
Hide comment
@mapazarr

mapazarr Jun 11, 2015

Member

@cdeil I am aware of the error (warning) in the build_sphinx in travis-ci. I'm working on it.
One question:
I have a file called:
docs/api/gammapy.datasets.generate_observation_table.rst
that issues a warning in my local build of sphinx. It comes from the time when make_test_observation_table was called generate_observation_table. But I don't remember creating the file myself. Can I just delete it? Or how do I clean all the references to the old name?

Member

mapazarr commented Jun 11, 2015

@cdeil I am aware of the error (warning) in the build_sphinx in travis-ci. I'm working on it.
One question:
I have a file called:
docs/api/gammapy.datasets.generate_observation_table.rst
that issues a warning in my local build of sphinx. It comes from the time when make_test_observation_table was called generate_observation_table. But I don't remember creating the file myself. Can I just delete it? Or how do I clean all the references to the old name?

@mapazarr

This comment has been minimized.

Show comment
Hide comment
@mapazarr

mapazarr Jun 12, 2015

Member

@cdeil I implemented the comments and fixed the warning in the sphinx doc. Could you please have a look at my concern in my previous comment?
Here is the new doc. BTW it doesn't take minutes to upload, buth rather ~ 1 h:
www.googledrive.com/host/0BzXHZQx0oCLBSGNKZDBZcnhlZzg/

Member

mapazarr commented Jun 12, 2015

@cdeil I implemented the comments and fixed the warning in the sphinx doc. Could you please have a look at my concern in my previous comment?
Here is the new doc. BTW it doesn't take minutes to upload, buth rather ~ 1 h:
www.googledrive.com/host/0BzXHZQx0oCLBSGNKZDBZcnhlZzg/

cdeil added a commit that referenced this pull request Jun 12, 2015

@cdeil cdeil merged commit 578a2b3 into gammapy:master Jun 12, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@cdeil cdeil changed the title from Generate observation tables to Document observation table format and improve gammapy.obs Jun 12, 2015

@cdeil cdeil changed the title from Document observation table format and improve gammapy.obs to Document observation tables and improve gammapy.obs Jun 12, 2015

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jun 12, 2015

Member

@mapazarr - Thanks for all your work on this and your first big PR for Gammapy being merged.

  • Forget about putting HTML docs online for code review then. I'll check them locally where needed.
  • I realise now that time handling in Gammapy is too complicated to discuss and address it as a side project in this PR. So I've created #284 and #285 where I'll continue this.

Probably the same will happen with (RA, DEC) <-> (ALT/AZ) <-> (DETX/DETY) coordinates.
It's complicated in detail, even if not much code needs to be written.
It's also needed for background modeling, which should be done in the tangential system and then background models reprojected onto sky images.
Please think about whether you want to learn about coordinates and take on the implementation and documentation of the FOV <-> Sky coordinate transformations and reprojection, or if that is something I should do. We can discuss and decide this in the Google hangout next week.

Member

cdeil commented Jun 12, 2015

@mapazarr - Thanks for all your work on this and your first big PR for Gammapy being merged.

  • Forget about putting HTML docs online for code review then. I'll check them locally where needed.
  • I realise now that time handling in Gammapy is too complicated to discuss and address it as a side project in this PR. So I've created #284 and #285 where I'll continue this.

Probably the same will happen with (RA, DEC) <-> (ALT/AZ) <-> (DETX/DETY) coordinates.
It's complicated in detail, even if not much code needs to be written.
It's also needed for background modeling, which should be done in the tangential system and then background models reprojected onto sky images.
Please think about whether you want to learn about coordinates and take on the implementation and documentation of the FOV <-> Sky coordinate transformations and reprojection, or if that is something I should do. We can discuss and decide this in the Google hangout next week.

@cdeil

This comment has been minimized.

Show comment
Hide comment
@cdeil

cdeil Jun 12, 2015

Member

Concerning your question about the old docs/api/gammapy.datasets.generate_observation_table.rst file, I've added an entry to the Gammapy developer docs here: https://github.com/gammapy/gammapy/pull/285/files#diff-00ab472a96028e9c3bcd77657d6c3701R15

Member

cdeil commented Jun 12, 2015

Concerning your question about the old docs/api/gammapy.datasets.generate_observation_table.rst file, I've added an entry to the Gammapy developer docs here: https://github.com/gammapy/gammapy/pull/285/files#diff-00ab472a96028e9c3bcd77657d6c3701R15

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment