D5.15: Final report and evaluation of all the GAP developments.

[minrk]

• WP5: High Performance Mathematical Computing
• Lead Institution: University of St Andrews
• Due: 2019-08-31 (month 48)
• Nature: Report
• Task: T5.2 (#100)
• Proposal: p. 51
• Final report (sources)

GAP ("Groups, Algorithms, Programming'') is a software system for computational discrete mathematics, with its origins in finite group theory. The system has been developed continually since 1986, with the first public release in 1988. Throughout its life it has been made available free of charge for use, extension and non-commercial redistribution. It is widely used by researchers in mathematics, computer science and other disciplines, and has been cited in more than 3000 research publications. It is also widely used in teaching. GAP is also used as a component of other mathematical software systems such as SageMath and OSCAR.

This report describes the work undertaken in connection with Task 5.2: a wide range of developments within the GAP system to meet the general objective of workpackage 5 ("improve the performance of the computational components of OpenDreamKit"). We have worked on supporting the use of a range of High Performance Computing environments, prioritizing those most accessible to our users, namely multi-core systems, adding to our existing support for clusters and cloud computing. We have also made fundamental performance improvements and improvements to our tools, processes and to the programming language, that benefit all GAP users.

A collection of GAP Jupyter notebooks, together with the setup to execute them on Binder, is available at https://github.com/OpenDreamKit/gap-demos.

[IzabelaFaguet]

Hello everyone!
We are organising an ODK report writing sprint From August 24th to August 31st,
a good opportunity to finish the final reports in a pleasant and friendly environment.
Would someone like to participate?

The link to the poll: https://framadate.org/tfuHjZgcSU8pHI45

[alex-konovalov]

@IzabelaFaguet thanks - we expect at least one of us from St Andrews will be able to come, but don't have a definitive decision yet.

[alex-konovalov]

What should be in the report (based on the notes from the whiteboard from the recent visit of @nthiery)

1. Developments in the core GAP system

   • libGAP (highlight)
   • other changes (see releases overview at https://www.gap-system.org/Manuals/doc/changes/chap0.html)

2. HPC-GAP

   • merging into mainstream GAP releases (see here)
   • meataxe64 interface (https://github.com/gap-packages/meataxe64 and https://meataxe64.wordpress.com/)

3. Packages

   • package manager
   • improving the health of the package ecosystem

4. Jupyter and derivatives

   • JupyterKernel
   • Francy
   • JupyterViz
   • other packages by @nathancarter

5. GAP distributions

   • Improved testing
   • Docker and other alternatives

6. Events

   • community building events
   • training events, Software Carpentry lesson, other workshops

7. Interfaces, WP6

   • high-level interoperability
   • persistent memoisation

8. A collection of demonstrators

   • "full-stack semigroups"
   • persistent memoisation
   • databases

[alex-konovalov]

@nthiery would you be able to import github descriptions for this please? Or this can be done later?

[nthiery]

Sure, will do. It can be done at any step later, so no rush; just edit the description on one side and the report on the other. Cheers,

[nthiery]

@nthiery would you be able to import github descriptions for this please? Or this can be done later?

Done! Let me know when you will have written an abstract in the github issue description, and I'll fetch it again. Cheers,

[nthiery]

Suggestion: please prepare a demo in notebook format; see #289.

[stevelinton]

@nthiery Noted. I have started discussion of this. We will aim to have something at least a couple of weeks before the review so that we can check all the binder setup etc.

[nthiery]

@nthiery Noted. I have started discussion of this. We will aim to have something at least a couple of weeks before the review so that we can check all the binder setup etc.

Great, thanks! If would be good to include a binder link to the notebook in the report when we submit it. So if we can have a stub notebook by the end of the week, or even just a decision of where exactly it will be eventually located, that's useful.

[stevelinton]

@nthiery When you have a moment could you have a look at the start of the introduction in the version I've just pushed (bottom of p2 and top of p3) and see if you think the general approach is plausible?

[alex-konovalov]

@nthiery wrote

Suggestion: please prepare a demo in notebook format; see #289.

I suggest that I will create a repository gap-binder-demos similar to https://github.com/alex-konovalov/gap-teaching under https://github.com/OpenDreamKit. Is that OK?

[nthiery]

I suggest that I will create a repository `gap-binder-demos` similar to https://github.com/alex-konovalov/gap-teaching under https://github.com/OpenDreamKit. Is that OK?

Sure thing. Could even be gap-demos since we will use the notebooks not only binder.

[alex-konovalov]

So be it! https://github.com/OpenDreamKit/gap-demos

[stevelinton]

@alex-konovalov @mtorpey I've added a Makefile. You don't have to use it, of course, but if you add any extra files (eg images), please update the Makefile.

[stevelinton]

@alex-konovalov @mtorpey Got to go do other things for a while. A few random thoughts becaus ethis is quicker than adding TODOs:

1. What are we going to do with the almost empty final sections on demos. Can we delete it, or just replace it with a line probably in the introduction that a selection of demonstrations of our achievements will be available at some URL from October 10th or something. We don't want to commit to specifics now and we certainly don't have time to debug run and include a lot of demos.

2. We should be consistent (at least mostly) on how we treat references to URLs. We could just have hyperlinks in the PDF, we could also have footnotes with the URLs or we could include the URLs in the running text (maybe in a slightly smaller font). personally I like footnotes. At the moment we have everything in different places.

3. Is section 5 about GAP package infratructure (seems to be) or where we report relevant developments in all packages (primarlity anything related to parallel computing or performance). If section 5 is the first one, where do we deal with the second one?

[nthiery]

2. We should be consistent (at least mostly) on how we treat references to URLs. We could just have hyperlinks in the PDF, we could also have footnotes with the URLs or we could include the URLs in the running text (maybe in a slightly smaller font). personally I like footnotes. At the moment we have everything in different places.

Reviewers might read a printed version, but anyway they are officially evaluating only what's in the report itself. Links are there for the convenience of other interested readers that would want to learn more. So I'd say hyperrefs are fine, unless we explicitly want to show the URL because it conveys some meaning.

[alex-konovalov]

1. Deleted that section, but keep Appendix A about https://github.com/sukru-yalcinkaya/unipoly (which is a stand-alone repo, not a part of the demo)

2. I prefer footnotes for URLs which are not so easy to google. E.g. Travis CI as href is fine, but a URL of the repository with some tool should be visible in a hard copy.

3. I created a new section with some list of packages. If looks OK, I will add them to the bibliography.

[stevelinton]

I've just been reviewing TODOs. We're very much getting there! The main general points:

1. I think we need to reorder the sections -- things obviously and directly relevant to WP5 and so HPC should be moved earlier; things that relate more to the general health of the ecosystem, or are basically just lists of other stuff that we did in the time period should be later. Then I can rewrite the overview accordingly.

Might be best to break the sections out into separate LaTeX files. Then reordering is just a change to a single file with \include statements in it.

2. In a similar vein, sections on testing, packaging infrastructure etc. need to constantly reiterate the point that this is relevant to High Performance Mathematical computing because it makes it easier for people to develop and share faster algorithms, or for some similar reason, or maybe say explicitly something like "to work on high performance mathematical computing in \GAP we needed to develop tool X which has turned out to have all kinds of other benefits as well,,,,".

In the office from 10.30ish.

[alex-konovalov]

@stevelinton @mtorpey thanks - I will do some maintenance tasks while you're in transit.

[alex-konovalov]

@stevelinton @mtorpey done - please pull new structure before doing anything.

[stevelinton]

Pushed a reordering (and mild restructuring), which, for me at least, puts the emphasis in the right places. Needs a pass to make sure that cross-references in text (as was discussed..., as will be discussed...) are still right, and that concepts are (usually) explained the first time they are encountered.

[ClementPernet]

@stevelinton @alex-konovalov : could you also update the progress report on Task T5.2 (which is essentially this deliverable) in section 1.2.5.2 of https://github.com/OpenDreamKit/OpenDreamKit/blob/master/ReportingPeriod3/TechnicalReport/WP5.tex
You can also just send me a paragraph and I'll edit the file.
Thanks

[stevelinton]

@ClementPernet I've pushed something. Feel free to adapt it as you see fit.

[jdemeyer]

While browsing though this report, I happened to notice that you're mentioning some GAP packages that have 0 contributors from ODK. Are we sure we should do that? After all, this report is supposed to be about work done thanks to ODK. I'm just asking the question, feel free to ignore me...

[alex-konovalov]

GAP got about 25-26 new packages during ODK, so that selection was carefully made to be connected to other developments described here. Just to indicate here that this was thought of.

[stevelinton]

I think we're clear about what we did and what we're just telling a story about. We're also emphasising work that enhanced support for the package ecosystem, so some package developments are relevant evidence of that.

[nthiery]

Yippee!
Thank you to the authors for the timely full-fledged report! And thank you everyone for all the improvements to GAP. Taking my Sage dev and user hat, I particularly appreciate everything you did to ease the integration in Sage.

[ClementPernet]

@stevelinton @alex-konovalov :
I am preparing the presentation of WP5. There are 4 deliverables due for M48 and this is one of them. Since it is the only deliverable about GAP, I think it will be important to highlight the achievements of this deliverable. Can you prepare a few slides (about 4-5) summarizing the main achievements in this deliverable, and the perspectives and new directions which follow from this work.
My slides for WP5 will be beamer slides, so you can

• prepare beamer slides
• or just list the content in plain text, and I can beamerize them.
  I will need to have this ready before sunday 27.
  Thanks.

[stevelinton]

Yes, we will do this.