Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PIG 12 - High-level interface #2219

Merged
merged 22 commits into from Aug 19, 2019

Conversation

@cdeil
Copy link
Member

commented Jun 6, 2019

This PR contains first ideas for a high-level interface for Gammapy.

Developing this is one of the projects on our roadmap (see here) that so far hasn't seen much discussion or developments.

Anyone with interest or time for this in the coming weeks or months, please consider contributing or taking the lead on this task!

This is a great way to learn about Gammapy, a lot of the work is wrapping and scripting the existing lower-level Gammapy functionality, for where there are already working examples in the tutorials: https://docs.gammapy.org/0.12/tutorials.html

@cdeil cdeil added the pig label Jun 6, 2019
@cdeil cdeil added this to the 0.13 milestone Jun 6, 2019
@Bultako

This comment has been minimized.

Copy link
Member

commented Jun 7, 2019

Hi all

I'll be happy to contribute in this one, of course I would need some help to define use cases and/or refine those existing as tutorials. I think one extra dimension that arises from this PIG is the possibility to capture structured logs of execution/session (provenance) so they could be easily queried in multiple ways later and inspected. This said, I would prefer to have separate PIG for prototyping provenance capture and inspection. The development of these PIGs would be more easy to do if we achieve a consistent and stable API for all/most of the use cases considered. Surely, one way to make this happen is start developing this High Level Interface, I only hope this will not require too much reorganization of the Gammapy code base... That's my big fear.

@cdeil cdeil requested review from adonath and registerrier Jun 7, 2019
@cdeil

This comment has been minimized.

Copy link
Member Author

commented Jun 7, 2019

1, 2, 3 ... sold to @Bultako!

Congratulations, you are now lead author for this PIG. Please add myself, and probaby @adonath and @registerrier also want to co-author. Feel free to push edits to this branch any time, I think it's simpler than to close and open a new PR.

I was also thinking about the scope for this PIG. Yes, config input and passing it down to other Gammapy code is super important. That probably needs to be in this PIG, although a first version of the high-level interface could be just scripting on top, with hard-coded config passed to class init and function calls, and then only later we develop a config system that's used throughout Gammapy.

Also provenance and logging is very important and related, and something that has to start with the high-level interface, but then be supported throughout Gammapy. For this I'd suggest to start by considering what ctapipe does, and also to look at https://eliot.readthedocs.io or what other science analysis pipelines do (e.g. JWST, LSST, ALMA, ...).

So you see, it's super vague at this point, but roughly I would suggest to either start with a small PIG here, or to already start a second small twin one in parallel concerning config / provenance / logging throughout Gammapy. My feeling is also that trying to put all of that into this one PIG would mean it becomes too large and unmanageable.

Ideally a first draft here would be developed by early July, extensively discussed in Erlangen, and by end of July there is an agreed plan what the high-level interface for Gammapy v1.0 is. @Bultako - Does that match your schedule, do you have time for this in the coming month?

@Bultako

This comment has been minimized.

Copy link
Member

commented Jun 7, 2019

Does that match your schedule, do you have time for this in the coming month?

Well, I'll have to !
No worries, I'll make some place in the agenda :)

Bultako and others added 3 commits Jun 13, 2019
@cdeil

This comment has been minimized.

Copy link
Member Author

commented Jul 16, 2019

Based on discussion this morning, here's the notes of what we tried to work out:

Bultako added 10 commits Jul 17, 2019
@cdeil cdeil modified the milestones: 0.13, 0.14 Jul 18, 2019
Copy link
Member Author

left a comment

@Bultako - Looks great. Thanks!

See inline comments.

docs/development/pigs/pig-012.rst Outdated Show resolved Hide resolved
docs/development/pigs/pig-012.rst Outdated Show resolved Hide resolved
docs/development/pigs/pig-012.rst Show resolved Hide resolved
docs/development/pigs/pig-012.rst Outdated Show resolved Hide resolved
docs/development/pigs/pig-012.rst Outdated Show resolved Hide resolved
docs/development/pigs/pig-012.rst Show resolved Hide resolved
docs/development/pigs/pig-012.rst Outdated Show resolved Hide resolved
Bultako added 2 commits Jul 18, 2019
@Bultako Bultako force-pushed the cdeil:pig-12 branch from e529523 to 3f74938 Jul 19, 2019
@Bultako Bultako force-pushed the cdeil:pig-12 branch 3 times, most recently from 22d1611 to c3bc37d Jul 19, 2019
@Bultako Bultako force-pushed the cdeil:pig-12 branch from c3bc37d to 61c665d Jul 20, 2019
Copy link
Member

left a comment

No further comments from my side. I think it's important to circulate this PIG early...

Copy link
Contributor

left a comment

Thanks @Bultako & @cdeil . Looks good.

Copy link
Member Author

left a comment

@Bultako - One small comment inline.

Actually, @registerrier @adonath and I realised that this proposal is basically ready to be circulated widely, and it should be soon, as that is very important for v1.0 to agree on what to build, and already for v0.14 we want to ship a first version. So could you please circulate it in the next days?
You can find examples on the Gammapy and CC mailing list for text to write - basically just a few sentences linking to the PIG and explaining where to give feedback. We think given that it's already summer, a deadline for comments of 3 weeks from now would be appropriate.

docs/development/pigs/pig-012.rst Show resolved Hide resolved
@Bultako Bultako dismissed stale reviews from registerrier and adonath via 477c153 Jul 26, 2019
@Bultako Bultako force-pushed the cdeil:pig-12 branch 2 times, most recently from b930831 to 3c7193e Jul 26, 2019
@Bultako Bultako force-pushed the cdeil:pig-12 branch from 3c7193e to 04fcd85 Jul 26, 2019
@cdeil

This comment has been minimized.

Copy link
Member Author

commented Aug 18, 2019

The comment period for this PIG is over (see https://groups.google.com/forum/#!topic/gammapy/hX8SjJUPCYM).

I think the suggestions by @bkhelifi have been taken into account?

If so, @Bultako - could you please mark the inline comments as resolved on Github, add one more commit to fill in the status / acceptance date in the header, and add a sentence in the decision section at the end (see other PIGs as examples), and then merge this in?

@Bultako Bultako merged commit 49ed630 into gammapy:master Aug 19, 2019
6 of 9 checks passed
6 of 9 checks passed
Codacy/PR Quality Review Hang in there, Codacy is reviewing your Pull request.
Details
Scrutinizer Running
Details
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
gammapy.gammapy Build #20190819.1 succeeded
Details
gammapy.gammapy (DevDocs) DevDocs succeeded
Details
gammapy.gammapy (Lint) Lint succeeded
Details
gammapy.gammapy (Test Python36) Test Python36 succeeded
Details
gammapy.gammapy (Test Windows36) Test Windows36 succeeded
Details
gammapy.gammapy (Test Windows37) Test Windows37 succeeded
Details
@cdeil

This comment has been minimized.

Copy link
Member Author

commented Aug 19, 2019

🎉 🎉 🎉 🎉 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.