Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
PIG 12 - High-level interface #2219
This PR contains first ideas for a high-level interface for Gammapy.
Developing this is one of the projects on our roadmap (see here) that so far hasn't seen much discussion or developments.
Anyone with interest or time for this in the coming weeks or months, please consider contributing or taking the lead on this task!
This is a great way to learn about Gammapy, a lot of the work is wrapping and scripting the existing lower-level Gammapy functionality, for where there are already working examples in the tutorials: https://docs.gammapy.org/0.12/tutorials.html
I'll be happy to contribute in this one, of course I would need some help to define use cases and/or refine those existing as tutorials. I think one extra dimension that arises from this PIG is the possibility to capture structured logs of execution/session (provenance) so they could be easily queried in multiple ways later and inspected. This said, I would prefer to have separate PIG for prototyping provenance capture and inspection. The development of these PIGs would be more easy to do if we achieve a consistent and stable API for all/most of the use cases considered. Surely, one way to make this happen is start developing this High Level Interface, I only hope this will not require too much reorganization of the Gammapy code base... That's my big fear.
1, 2, 3 ... sold to @Bultako!
Congratulations, you are now lead author for this PIG. Please add myself, and probaby @adonath and @registerrier also want to co-author. Feel free to push edits to this branch any time, I think it's simpler than to close and open a new PR.
I was also thinking about the scope for this PIG. Yes, config input and passing it down to other Gammapy code is super important. That probably needs to be in this PIG, although a first version of the high-level interface could be just scripting on top, with hard-coded config passed to class init and function calls, and then only later we develop a config system that's used throughout Gammapy.
Also provenance and logging is very important and related, and something that has to start with the high-level interface, but then be supported throughout Gammapy. For this I'd suggest to start by considering what ctapipe does, and also to look at https://eliot.readthedocs.io or what other science analysis pipelines do (e.g. JWST, LSST, ALMA, ...).
So you see, it's super vague at this point, but roughly I would suggest to either start with a small PIG here, or to already start a second small twin one in parallel concerning config / provenance / logging throughout Gammapy. My feeling is also that trying to put all of that into this one PIG would mean it becomes too large and unmanageable.
Ideally a first draft here would be developed by early July, extensively discussed in Erlangen, and by end of July there is an agreed plan what the high-level interface for Gammapy v1.0 is. @Bultako - Does that match your schedule, do you have time for this in the coming month?
cdeil left a comment
@Bultako - One small comment inline.
Actually, @registerrier @adonath and I realised that this proposal is basically ready to be circulated widely, and it should be soon, as that is very important for v1.0 to agree on what to build, and already for v0.14 we want to ship a first version. So could you please circulate it in the next days?
The comment period for this PIG is over (see https://groups.google.com/forum/#!topic/gammapy/hX8SjJUPCYM).
I think the suggestions by @bkhelifi have been taken into account?
If so, @Bultako - could you please mark the inline comments as resolved on Github, add one more commit to fill in the status / acceptance date in the header, and add a sentence in the decision section at the end (see other PIGs as examples), and then merge this in?