Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PIG 18: Documentation #2463

Merged
merged 15 commits into from Nov 6, 2019
Merged

PIG 18: Documentation #2463

merged 15 commits into from Nov 6, 2019

Conversation

@cdeil
Copy link
Member

cdeil commented Oct 15, 2019

This is a proposal to spend a lot of effort in the next 2 month to improve the Gammapy documentation. We outline the desired state and required work, but do it very briefly. Each of the 10-20 tasks will then be implemented via Github issues & pull requests where the content will be discussed and reviewed in detail.

The timeline here is that this is a short and focused effort: @adonath @Bultako and I plan to write this PIG in the next days, and then we circulate it for a week to gather feedback, and then we have ~ one month, Nov 2019, to implement as many improvements as we can. The hope is that if there is an "agreed plan" and "task list" what to do, the work itself will progress quickly and with a few people working on the documentation in parallel.

@cdeil cdeil added docs pig labels Oct 15, 2019
@cdeil cdeil added this to the 0.15 milestone Oct 15, 2019
@cdeil cdeil requested review from Bultako and adonath Oct 15, 2019
@cdeil cdeil self-assigned this Oct 15, 2019
@cdeil cdeil added this to To Do in DOCUMENTATION via automation Oct 15, 2019
cdeil and others added 2 commits Oct 16, 2019
@adonath adonath force-pushed the cdeil:docs-pig branch from ed04e84 to 9cc3978 Oct 17, 2019
adonath and others added 4 commits Oct 17, 2019
@cdeil

This comment has been minimized.

Copy link
Member Author

cdeil commented Oct 18, 2019

Current proposal draft: https://github.com/gammapy/gammapy/blob/1c5cc1ea164402cf46dd4375165af79eedcfabd9/docs/development/pigs/pig-018.rst

Comments very welcome today!
I plan to circulate widely tonight, so we should polish as much as we can.

Specifically, @adonath @registerrier @Bultako @cboisson or anyone interested - Please comment on these questions:

  • Should we keep or remove the two Sherpa tutorial notebooks?
  • For CWT, should we completely remove the code (quality not good enough, never debugged), or shorten the extra notebook https://docs.gammapy.org/0.14/notebooks/cwt.html to a short section in a detect notebook, or keep the CWT notebook as-is in the extra topics section at the end?
  • For HGPS tutorial, should we keep it in the extras list and I shorten it to ~ half the size? Or should we completely remove it, and have HGPS only mentioned in the catalog tutorial with one small example?

My 2 cents:

  • Would remove Sherpa tutorials (and Sherpa from our env file) to reduce maintenance effort. Their quality is OK, but not great (e.g. they should show exact same results as what can be done with Gammapy). But I guess some people are strongly +1 to keep them?
  • For CWT, would remove completely. But OK to keep in extras section as-is or polish a bit if someone has time.
  • For HGPS, would keep and shorten to half the length. I think it's a nice example how show how to work with survey maps and do plotting. And HGPS is a specifically complex catalog, with the multi-Gauss and diffuse band model, it's not well-suited for a short section in a general catalog tutorial.
Bultako added 2 commits Oct 19, 2019
@Bultako

This comment has been minimized.

Copy link
Member

Bultako commented Oct 19, 2019

I'm +1 to keep the three notebooks as long as there is no duplicated concepts/recipes also present in other notebooks. If they are too specific we can always put them in the Extra topics section, but in general I'm against throwing away info that could be potentially useful for others. I would take the decision on what to remove and what to keep from those three notebooks during the process of defining the details of the others.

More comments:

  • I'd put some effort in revisioning the completeness and consistency of the API docstrings.
  • In general the notebooks should use high-level interface everywhere it makes sense (i.e. automatic data reduction), and the lower-level API at the end for the very specific use case proposed, so to have shorter notebooks going to the point.
  • I'd add a HowTo list of needs linking to subsections of notebooks showing the solution. I'd propose to write this HowTo list of needs independently of the content of notebooks, which could help us to understand how complete or useful are our tutorials. A very first proposition of what this HowTo should cover below:

Modeling

  • How to test the spectral and/or spatial model
  • How to calculate lower/upper limits for a spectral parameter

SED

  • How to calculate integral/differential flux and upper limits
  • How to calculate spectral points and residuals
  • How to plot the SED with errors

Source Detection

  • How to build and display the on region
  • How to get the significance (Li&Ma, ΔTS)
  • How to get excess and its error
  • How to get background counts

2D Morphology

  • How to define/get position and spatial dimensions at different energy thresholds
  • How to calculate surface brightness or radial profile in within a specific mask/region
  • How to calculate a spectrum within a specific mask/region
  • How to overlay significance and excess on maps

Light Curves

  • How to do analysis of light curves and upper limits
@cdeil

This comment has been minimized.

Copy link
Member Author

cdeil commented Oct 23, 2019

@Bultako - Thanks for the suggestions.

About the suggestion to add a HOWTO page -- maybe we can discuss this a bit and try to flesh out the idea this week, so that we can update the PIG with a concrete and mostly agreed addition early next week. @adonath @registerrier @cboisson or anyone interested - please chime in.

I think we should add something like a HOWTO or FAQ page, but it's not obvious what we should and shouldn't add there. E.g. @Bultako - of the possible items you add, I think "how to get significance" would be a good one, because it's not a tutorial in itself, but a few tutorials have sections that show how to do this, so we can link to them. But "how to do an analysis of light curves" is just a tutorial notebook, I'm not sure if we should add this to the HOWTO list - basically we'd have to add 10 - 20 entries that match notebooks that show a given task.

Examples of what others have:

@Bultako - I guess what you have in mind is a single page howto.rst where each entry is one or a few lines of text with references? Maybe the simplest and quickest way to get it would be if you start a PR now with 2-3 examples of entries (or paste your list from the comment above as starting point), and then we just reference that PR from the PIG.

I'm +1 to add something like that, whether it's better called HOWTO or FAQ I don't know.

@cdeil

This comment has been minimized.

Copy link
Member Author

cdeil commented Oct 23, 2019

Another question is if we should have a glossary.

At the moment we have one here, with one entry:
https://docs.gammapy.org/0.14/references.html#glossary

I'm +1 to have a glossary and to reference it throughout when using some non-obvious term like "GTI", "MET", "OBS_ID", "PSF", ...

It's not hard to implement, we start with ~ 10 relevant items from existing glossaries and then start referencing it when working on the Gammapy docs:

Thoughts?

@adonath

This comment has been minimized.

Copy link
Member

adonath commented Oct 23, 2019

👍 For the glossary.

@registerrier

This comment has been minimized.

Copy link
Contributor

registerrier commented Oct 23, 2019

In general, I think that a documentation relying heavily on notebooks might be an issue.

Most of users tend to copy paste the code from the notebooks to build their own analysis. It leads to errors difficult to track in many cases, e.g. different column names in the obs_table, improper definition of energy ranges, etc.

As clearly written in the PIG, we propose both a science tools package and a flexible python library. I think this dichotomy should be reflected in the documentation.

RST files are therefore really important to properly explain the code structure. They are also needed to detail the methodology for data reduction and fitting. Notebooks should always refer to them to explain what is being done.
In a sense there should be RST pages, which do not really belong to a specific package but detail general notions on data analysis and how these are handled in gammapy.

@cdeil

This comment has been minimized.

Copy link
Member Author

cdeil commented Oct 23, 2019

@registerrier - Can you make specific suggestions for RST docs you'd add, or suggest edits to the General Concept section?

Note the you can make a PR against this PR with your proposal, or just write your proposal into a https://gist.github.com/ or long comment here for discussion.

We're already proposing to add an overview.rst. And we're not proposing to delete the RST pages for modeling, maps, ..., the plan as stated is that they remain, and just are complimentary to the notebooks instead of duplicating content (like we currently have for maps).

Maybe add a datasets.rst page which describes DL3 data -> data reduction -> datasets instead of or in addition to the proposed new datasets.ipynb notebook?

As clearly written in the PIG, we propose both a science tools package and a flexible python library. I think this dichotomy should be reflected in the documentation.

How?

Are you proposing a completely different and better way to structure the docs, consisting of two separate parts?

Or do you think adding 2-3 extra RST pages to the existing docs is the way to go?

@adonath

This comment has been minimized.

Copy link
Member

adonath commented Oct 23, 2019

@registerrier We definitely want to add an overview RST page explaining the details of data analysis and give an overview about concept such as Datasets, Fit, Models etc. and how those play together in a Gammapy analysis. Note that you can also download notebooks as python scripts already e.g. https://docs.gammapy.org/0.14/_static/notebooks/first_steps.py.

But in general I agree, if we put the focus of the documentation on notebooks, the users gain the impression it is the only valid way to use Gammapy. However notebooks are of course just one way of using Gammapy, that is suitable for teaching / tutorials. Maybe we should add a section in the PIG with a proposal how to improve and document others usage of Gammapy such as scripts and / or IPython?

@AtreyeeS

This comment has been minimized.

Copy link
Member

AtreyeeS commented Oct 28, 2019

The PIG looks quite good to me. A few comments

  1. Perhaps pulsed analysis should go in the section What analyses can I do?
  2. Have a light curve simulation tutorial.
  3. I guess all the data reduction steps will be done through the high level interface in all the notebooks? Since for standard analysis the fitting can be done through the Analysis class, I guess the IACT 3D cube analysis and IACT 1D spectrum analysis will also be config driven? I think it will be nice to explain to users how to access the datasets and do customised fittings (eg: a modified likelihood) or just dig deep to make debug plots.
  4. I don't know if it is really useful, but maybe we should also have one notebook detailing the data reduction steps (just to make it clear what happens in the high level interface, instead of having it completely under the hood).
  5. I really like @Bultako suggestion to have several little HOWTO snippets. This addresses not just the coding part, but also explains the science part of it... It would be nice to access a significance calculation directly instead of searching through 10 notebooks.
  6. As @registerrier mentioned, users will probably copy-paste code from the notebooks a lot. Perhaps we should detail the markdown sections much more (eg: Mention the values the user should adapt, or the default parameters that can be passed, or the reason we choose certain values (to get enough statistics/ to have fine enough bins/ etc) so that its apparent what can be reasonably tuned for different sources)
@cdeil

This comment has been minimized.

Copy link
Member Author

cdeil commented Oct 28, 2019

@AtreyeeS - Thanks for the comments!

Deadline for comments is tonight:
https://groups.google.com/forum/#!topic/gammapy/uvcAPEoGyR8

@registerrier @Bultako @adonath - if you have concrete suggestions or further comments, please put them today.

I'll then do a big update to the PIG tomorrow, trying to take all comments received into account.

@cdeil cdeil moved this from To Do to In Progress in DOCUMENTATION Oct 29, 2019
@cdeil cdeil assigned Bultako and unassigned cdeil Oct 31, 2019
@cdeil

This comment has been minimized.

Copy link
Member Author

cdeil commented Oct 31, 2019

@Bultako @adonath and I had a call on documentation this morning.

Added an "Overview" and "How To" page, already done in #2500.
It's empty for now, filling it will be done in the coming weeks, PRs welcome any time.

Change the "DOCUMENTATION" Github project (see https://github.com/gammapy/gammapy/projects/1) to be only concerned with documentation content.
The idea is to fill this board to identify the 10 - 20 tasks / pull requests we want for documentation in the coming weeks, and then try to distribute the work a bit. For this, @cdeil will try to find a coordinator next week. Doing that work will then be a concerted effort for the next 6 weeks, long-term discussions and suggestions how to completely rework the docs in 2020 and beyond are not productive at this time.

Added "MAINTAIN" Github project (see https://github.com/gammapy/gammapy/projects/16) that contains things like Sphinx + Jupyter + other tooling issues. The goal was to have the DOCUMENTATION board be simple and focused on docs content only, so that the coordinator and possibly new contributors could quickly see how it works and pick up tasks. Whether to use "notes" or "issues" for each docs task will be discussed next week, and will be basically up to the documentation coordinator.

@Bultako volunteered to update and finalise this PIG early next week, taking the comments received above into account (there were no others, e.g. on the mailing list or elsewhere). The goal should be to keep the PIG short and do this quickly, e.g. simply by linking to exiting docs/howto.rst or docs/overview.rst or DOCUMENTATION project board, instead of detailing things in the PIG. The advantage of this is to go as fast as possible towards getting it done. @Bultako @adonath and I agreed that while there were significant comments on this PIG, and the discussion was super useful, we would not do another official review and circulation. There's basically agreement on what to do for the docs, and we should just do it and iterate towards good docs, without wasting more weeks on discussing before starting.

@cdeil

This comment has been minimized.

Copy link
Member Author

cdeil commented Nov 4, 2019

list of documentation examples of projects

It's in the PIG. I think it'll be useful to have these examples as reference, also in the future?
So I'm slightly +1 to keep it in the PIG. It's super unlikely that people in the future would find and read a comment on a closed PR on Github like #2463 (comment) .

Bultako added 2 commits Nov 4, 2019
@Bultako Bultako force-pushed the cdeil:docs-pig branch 2 times, most recently from a9e1161 to 23e45ca Nov 4, 2019
@Bultako

This comment has been minimized.

Copy link
Member

Bultako commented Nov 4, 2019

@cdeil @adonath
from my side this is ready to merge.

@Bultako Bultako force-pushed the cdeil:docs-pig branch from 23e45ca to 600302e Nov 5, 2019
@cdeil

This comment has been minimized.

Copy link
Member Author

cdeil commented Nov 6, 2019

@Bultako - Thanks!

I'll do some more polishing for this PIG today and then merge it in.
I'll also spend some time to fill https://github.com/gammapy/gammapy/projects/1 and prepare the documentation work tomorrow.

@cdeil cdeil assigned cdeil and unassigned Bultako Nov 6, 2019
@cdeil

This comment has been minimized.

Copy link
Member Author

cdeil commented Nov 6, 2019

@Bultako - Here's the update I did: 0826831

Personally I don't agree exactly which notebooks should be in the main "what can I analyse" and "extra topics" section as written here. But I don't think it's productive to argue about that now, we should just get going and improve the docs iteratively now. I've left a sentence in the decision section stating that the outline here is a rough plan, which will be adjusted based on what contributions and feedback we get and learn in the coming weeks and beyond.

@cdeil

This comment has been minimized.

Copy link
Member Author

cdeil commented Nov 6, 2019

@Bultako @adonath @registerrier - can you please have a final look, and if OK with you approve this PR? (or if you want, do some further improvement directly or leave a suggestion in a comment in the next hours)

@Bultako
Bultako approved these changes Nov 6, 2019
@cdeil cdeil merged commit 7d06f74 into gammapy:master Nov 6, 2019
9 checks passed
9 checks passed
Codacy/PR Quality Review Up to standards. A positive pull request.
Details
Scrutinizer Analysis: No new issues – Tests: passed
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
gammapy.gammapy Build #20191106.6 succeeded
Details
gammapy.gammapy (DevDocs) DevDocs succeeded
Details
gammapy.gammapy (Lint) Lint succeeded
Details
gammapy.gammapy (Test Python36) Test Python36 succeeded
Details
gammapy.gammapy (Test Windows36) Test Windows36 succeeded
Details
gammapy.gammapy (Test Windows37) Test Windows37 succeeded
Details
DOCUMENTATION automation moved this from In Progress to Done Nov 6, 2019
@cdeil

This comment has been minimized.

Copy link
Member Author

cdeil commented Nov 6, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
DOCUMENTATION
  
Done
5 participants
You can’t perform that action at this time.