Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

D3.10: Packaging components and user-contributed code for major Linux distributions #59

Closed
minrk opened this issue Sep 8, 2015 · 44 comments
Closed

Comments

@minrk
Copy link
Contributor

@minrk minrk commented Sep 8, 2015

This deliverable addresses the following objectives of OpenDreamKit:

Objective 1: To develop and standardise an architecture allowing combination of mathematical data and software components with off-the-shelf computing infrastructure to produce specialised VRE for different communities.

Objective 3: To bring together research communities (e.g. users of Jupyter, GAP, PARI, Sage, and Singular) to symbiotically exploit overlaps in tool creation efforts, avoid duplication of effort in different disciplines, and share best practice.

Objective 4: Update a range of existing open source mathematical software systems for seamless deployment and efficient execution within the VRE architecture of objective 1.

Objective 5: Ensure that our ecosystem of interoperable open source components is sustainable by promoting collaborative software development and outsourcing development to larger communities whenever suitable.

We contribute to the achievements of these objectives through the creation of source and binary packages for major Linux distributions, for all OpenDreamKit components.

Sage has a long history of integrating and distributing large mathematical libraries/software as a whole, with relatively little attention given to defining and exposing interfaces. This conscious compromise was made by the Sage community during the first stages of the project, in order to reach as quickly as possible the critical mass required for its survival. Component re-usability was then not the main focus for the Sage community; at the same time the non-standard and relatively underused package system discouraged writing and maintaining autonomous libraries. These factors contributed to make the Sage distribution what is usually described as a “monolith” (Sage library code alone, not counting included libraries, makes up for 1.5M lines of code and documentation), hard to distribute, to maintain, to port, and to develop with as it reached maturity. On the other hand, GAP has been distributing community-developed “GAP packages” for a long time, but faced fragmentation issues, at the code and at the community level. The rudimentary package system added more technical difficulties to GAP’s development model.

We achieved the stated goal of packaging for major Linux distributions through several actions:

  • Workshops dedicated to packaging,
  • Limiting patched dependencies in OpenDreamKit software,
  • Updating dependencies of OpenDreamKit software,
  • Modularization of OpenDreamKit software,
  • Providing alternate workflows for user-contributed code, thanks to system-specific packaging tools and repositories.
  • Contributing to packaging OpenDreamKit components, new or pre-existing, for cross-platforms packaging systems.
@minrk minrk added this to the D3.10 milestone Sep 8, 2015
@nthiery nthiery mentioned this issue Sep 11, 2015
3 of 3 tasks complete
@minrk minrk mentioned this issue Nov 1, 2015
1 of 7 tasks complete
@nthiery
Copy link
Contributor

@nthiery nthiery commented Nov 1, 2015

One plan to get started on this would be to have a dedicated workshop (typically in Orsay or nearby Cernay next Spring). Here are some tentative names of people from ODK that will work on this and people that have been involved in packaging Sage & co (please expand): Erik Bray, Julien Cristau (Debian/Logilab), Jan Groenewald, Thierry Monteil, François Bissey, Burcin Erocal, ...

@fangohr
Copy link
Contributor

@fangohr fangohr commented Nov 2, 2015

Sounds good - we may want to send somebody from Southampton to join the effort (with focus on packages the micromagnetic package(s)).

@alex-konovalov
Copy link
Member

@alex-konovalov alex-konovalov commented Nov 2, 2015

Sounds good - hope GAP will be also represented.

@jcristau
Copy link

@jcristau jcristau commented Nov 5, 2015

Sounds good to me.

@jdemeyer
Copy link
Contributor

@jdemeyer jdemeyer commented Nov 5, 2015

I'm also interested to discuss things. I must say that I am pessimistic about the "packaging for distributions" effort, so it really would be useful to have somebody who represents the distributions.

@defeo
Copy link
Contributor

@defeo defeo commented Nov 12, 2015

Some reading on what's happened recently in Fedora viz bundling http://lwn.net/Articles/660429/.

Some discussion here: http://lwn.net/Articles/660429/. Note the mention of Guix in the first comment.

@embray
Copy link
Collaborator

@embray embray commented Feb 29, 2016

I've started some side work on my own list of Sage's dependencies, and how they relate to Debian. So far my finding is that the vast majority of dependencies can be satisfied through the system packager in that case. The improvements needed on Sage's end, then, are to determine which dependencies can and can't be satisfied from the system (in some cases they can't due to unique patches provided by Sage, but in many other cases the only patches are unique to building that package within Sage, and not relevant if it comes from the system package).

@nthiery nthiery modified the milestones: Month 48: 2019-08-31, D3.10 Mar 22, 2016
@slel
Copy link
Contributor

@slel slel commented Jul 13, 2016

Recently read on Sage-devel at
https://groups.google.com/d/msg/sage-support/e91qG-qBGAY/ZXKmYpo3CgAJ

Ideally each optional package should be in itself
a debian package. But we lack manpower...

@videlec
Copy link
Contributor

@videlec videlec commented Aug 8, 2016

@embray do you know of (the recently edited) https://wiki.debian.org/DebianScience/Sage

@fcayre
Copy link

@fcayre fcayre commented Aug 9, 2016

@videlec, @embray there are undergoing discussions between Logilab and Jérôme Benoît (calculus@rezozer.net) to see if Logilab will hire him to help packaging Sage in Debian. I'll keep you informed on this thread.

@jdemeyer
Copy link
Contributor

@jdemeyer jdemeyer commented Aug 9, 2016

Good news. I hope that we can work with the Debian people to actually improve Sage to make it more distributable. That will benefit not just Debian, but all distros. We already added a few patches from Gentoo.

@embray
Copy link
Collaborator

@embray embray commented Aug 9, 2016

@embray do you know of (the recently edited) https://wiki.debian.org/DebianScience/Sage

I know of that page, but I didn't know it was recently updated. Looks like it's been significantly cleaned up since I last saw it.

@slel
Copy link
Contributor

@slel slel commented Aug 9, 2016

@slel
Copy link
Contributor

@slel slel commented Aug 10, 2016

A dedicated mailing list "debian-science-sagemath" was just created.

@serge-sans-paille
Copy link
Contributor

@serge-sans-paille serge-sans-paille commented Aug 19, 2016

If as a side effect, pythran could be packaged for debian, that would be great :-)

@dimpase
Copy link
Contributor

@dimpase dimpase commented Jan 22, 2017

I wish one could channel debian-science activity more towards improving Sage, instead of at times being the cart before the horse; e.g. they updated libGAP to GAP 4.8.6 on their own debianised fork, and we only now found out about this as they started to ask about failing Sage doctests.

@jgmbenoit
Copy link

@jgmbenoit jgmbenoit commented Jan 22, 2017

@dimpase
Copy link
Contributor

@dimpase dimpase commented Jan 22, 2017

@jgmbenoit: "Sage-centric?" Not at all. libGAP has a very well-defined upstream to which you chose not to contribute for a reason I fail to comprehend.

@nthiery
Copy link
Contributor

@nthiery nthiery commented Jan 22, 2017

@jgmbenoit
Copy link

@jgmbenoit jgmbenoit commented Jan 23, 2017

@dimpase
Copy link
Contributor

@dimpase dimpase commented Jan 23, 2017

@nthiery
Copy link
Contributor

@nthiery nthiery commented Aug 30, 2019

On behalf of @defeo: the write up is half way through. The currently written sections should be fairly stable. Proofreading and feedback on them very welcome (@embray?). I'll be on the road until 8pm, so the file is completely fair game.

@defeo
Copy link
Contributor

@defeo defeo commented Sep 2, 2019

This deliverable is ready for proofreading, sorry for the delay.

@embray , @alex-konovalov , @KBelabas , @wbhart @ClementPernet , please tell me if I forgot to mention some important facts / people related to pacakging.

Also, the author list for the deliverable has to be cleaned-up. @nthiery, should we list only those who contributed to writing the deliverable, or everyone involved in packaging? In the second case, I may have missed some people.

@nthiery
Copy link
Contributor

@nthiery nthiery commented Sep 2, 2019

@wbhart
Copy link
Contributor

@wbhart wbhart commented Sep 2, 2019

@defeo I don't see any changes since the last time I proofread it. Elsewhere I happened to notice another typo:

Were -> We

@defeo
Copy link
Contributor

@defeo defeo commented Sep 2, 2019

@defeo I don't see any changes since the last time I proofread it. Elsewhere I happened to notice another typo:

Were -> We

@wbhart , there is no "Were" in the text. Are you confusing with D3.11?

@wbhart
Copy link
Contributor

@wbhart wbhart commented Sep 2, 2019

@defeo Possibly. But that's what's attached at the link above. Where is the report you want me to review?

@defeo
Copy link
Contributor

@defeo defeo commented Sep 2, 2019

@defeo Possibly. But that's what's attached at the link above. Where is the report you want me to review?

Oh, I see. The links were pointing to the wrong deliverable. I corrected them.

@wbhart
Copy link
Contributor

@wbhart wbhart commented Sep 2, 2019

Since long -> For a long time
accustomed its users with -> accustomed its users to
by running it from external media (e.g., a CD or DVD) :: this is no longer common, in fact most modern desktops do not have optical drives and almost everything is exclusively offered online
Julia General Pkg2 :: The Julia package manager is Pkg. The 3 (not 2) is the current version.
a OS-and-language-agnostic -> an OS-and-language-agnostic
if two software have -> if two software systems have
not under our direct responsibility -> not our direct responsibility
the vast majority of which runs -> the vast majority of which run
package system for a OpenDreamKit software, in the same vein of language-specific package systems -> package system for OpenDreamKit software, in the same vein as language-specific package systems
(footnote) like for Anaconda -> for example Anaconda OR as with Anaconda
means that a software -> means that software
table 4 :: might benefit from [htd]
Information on the packages available at the time -> Information on the packages available at the time of writing ??
dependency tree of each software -> dependency tree of each software package
fell off the official distribution -> fell out of the official distribution
If a software can be installed on a system -> If a software package can be installed on a system
The difficulty is not packaging a software -> The difficulty is not packaging software
shared by several software -> shared by several software packages
needs only be loaded -> needs to only be loaded
different versions of a same component -> different versions of the same component
some times it is impossible -> sometimes it is impossible
version of a component, in this case -> version of a component. In this case
side to side -> side-by-side
a software may rely on -> software may rely on
which component he prefers to have installed, and even change his mind later :: gender neutrality please
complexity of packaging a software -> complexity of packaging software
a single software may interact -> a single software package may interact
In this sense :: not a good way to start a new paragraph
dependencies nearly :: perhaps omit nearly, as I don't understand what its function is in this sentence
A series of workshop -> A series of workshops
Failing which :: not a new sentence
Finally, a different issue, mostly :: there is a line overrun
that allow to easily share -> that facilitate easy sharing
no designed maintainer -> no designated maintainer
work spent -> effort expended OR work undertaken
efforts done -> effort undertaken
tasks of package maintainers -> workflow of package maintainers ??
Binder-compatbile -> Binder-compatible
self contained -> self-contained
SINGULAR is in the process of migrating to Julia, in the context of the OSCAR project, and will likely use the standard Julia package tools when the migration will be complete :: not true, SINGULAR will continue to exist in its current form, with a Singular -> Julia transpiler and Singular.jl package wrapping libSingular in the works. PLEASE REMOVE THIS as it will be misleading to the Singular community.

[software in English cannot be made singular or plural without adding a word which has a singular or plural form. It is a mass noun, like furniture, knowledge, bravery, wisdom, help, etc.]

@defeo
Copy link
Contributor

@defeo defeo commented Sep 2, 2019

Thanks for the corrections @wbhart. Document updated.

@embray
Copy link
Collaborator

@embray embray commented Sep 4, 2019

I see there's a report-final.pdf in the repository for this already so I'm not sure if it's too late or not. But aside from some (very minor, not serious) typos, I have a minor quibble with the conclusion that Python 3 support will make anything easier regarding conda packages for Windows. I'm not really sure where that idea comes from (unless we're not talking about conda packages for Sage itself...?)

@defeo
Copy link
Contributor

@defeo defeo commented Sep 4, 2019

I see there's a report-final.pdf in the repository for this already so I'm not sure if it's too late or not.

I don't think @nthiery has submitted the report, there should still be time to fix it.

But aside from some (very minor, not serious) typos, I have a minor quibble with the conclusion that Python 3 support will make anything easier regarding conda packages for Windows. I'm not really sure where that idea comes from (unless we're not talking about conda packages for Sage itself...?)

No, I guess that's wishful writing. I haven't followed closely packaging for Conda, I'm not sure what the problems for Windows are, but I assume you know, so if you want to write a few paragraphs about it, that would be very helpful.

At least, I believe it is true that moving to Py3 will make packaging for Conda easier, regardless of Linux or Windows. Isn't it?

@embray
Copy link
Collaborator

@embray embray commented Sep 4, 2019

Well, long-term yes, simply because more Python-based dependencies will become more and more Python 3-only over time. But I don't think there's anything specific that that helps on Windows, unless I'm missing something.

The problem with Windows is that many of Sage's dependencies still cannot work without Cygwin, and there is as of yet a way to distribute packages on conda with a Cygwin dependency. That's something I've been wanting to do, but I believe it's a significant-enough effort that it would take some time, and close collaboration with the Anaconda people (if they even wanted to support that, which would be cool if they did...)

@nthiery
Copy link
Contributor

@nthiery nthiery commented Sep 4, 2019

Indeed: I did not yet get to submit the report, and probably won't until tomorrow late morning. Feel free to do some editing by then.

@nthiery
Copy link
Contributor

@nthiery nthiery commented Sep 5, 2019

Proofreading. Questions:

  • Linux or GNU/Linux (I tend to prefer the second one but don't want to go into religion :-) )?
  • Anaconda or Conda (to name package system)?
@dimpase
Copy link
Contributor

@dimpase dimpase commented Sep 5, 2019

Conda, not Anaconda

(as far as Linux is concerned, doesn't matter)

I'm back to the office, can work on the reports in the coming days.
Please direct me to the right tasks, it seems something has already been submitted...

@nthiery
Copy link
Contributor

@nthiery nthiery commented Sep 5, 2019

Review finished. I did minor updates here and there, and in the github issue description. I will submit around 2pm.

@nthiery
Copy link
Contributor

@nthiery nthiery commented Sep 5, 2019

Dima: thanks for the offer! If you can proofread my latest change, that's nice. Will contact you for additional tasks.

@nthiery
Copy link
Contributor

@nthiery nthiery commented Sep 5, 2019

I did a few minor edits, mentioned GNU/Linux, mentioned the packaging efforts beyond Linux (conda) in the abstract, and mentioned the packaging of most of the new software we implemented in the text itself.

And submitted!

I believe this gives a very convincing account of the massive efforts undertaken by ODK and the community about modularization and ease of distribution, eventually making a huge difference for end-users and for the sustainability of our software.
Thank you everyone for those efforts! And thank you Luca for the report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet