Fortran needs packaging ecosystem

[certik]

Most other languages have that, whether Python, Julia, Go, Rust, JavaScript....

Goals:

• Make it easy to create an application that depends on N external libraries written in Fortran, and the build system can easily and robustly build it

• Make it easy to create a new Fortran package (that depends on other packages) and distribute it

• Package index where people can easily search for available packages provided by other people

• Build a community and culture of such packages

• Must work with any Fortran compiler

This needs to be carefully designed, we need to learn from the above mentioned languages.

Some related projects to consider:

• Spack, unfortunately does not work on Windows (Windows Support spack/spack#9323)
• Conda and conda-forge
• Julia package manager

Note: initially opened at https://gitlab.com/lfortran/lfortran/issues/109.

[zmiimz]

imho, one of the best to consider is the dub manager from dlang project (it is also AIO: package manager and build system)
https://github.com/dlang/dub

[septcolor]

FWIW, this site gathers statistics for package registries of various languages. We can see more details by clicking the name of the registries.
http://www.modulecounts.com/

[sblionel]

Looks like all the languages mentioned are interpreted. Keep in mind that there's more to the Fortran world than Windows, Linux and Mac. The Fortran world seems to have gotten along well with libraries without something in the standard for a package manager. These sorts of things tend to go obsolete quickly, anyway. I don't see it as something appropriate to add to the standard, which doesn't even discuss what source file names look like.

[gronki]

While it might not be a good thing to put in the standard per se, it's definitely something that needs initiative from the standarization side. For example, while the format of module files has never been standarized (probably for convenience of compiler vendors and annoyance of users), distributing Fortran packages is hell even in the current scope (for example withit one Linux distribution). So while the system itself is not dependent on the standard, to my knowledge the standard in its current form does not make it possible to build one. niedz., 3 lis 2019 o 21:43 sblionel <notifications@github.com> napisał(a):

…

Looks like all the languages mentioned are interpreted. Keep in mind that there's more to the Fortran world than Windows, Linux and Mac. The Fortran world seems to have gotten along well with libraries without something in the standard for a package manager. These sorts of things tend to go obsolete quickly, anyway. I don't see it as something appropriate to add to the standard, which doesn't even discuss what source file names look like. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#55?email_source=notifications&email_token=AC4NA3LT7MKBKSCVTWJORYLQR4SW5A5CNFSM4JFTVA7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC54JWA#issuecomment-549176536>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC4NA3NDONJH74U2VLST45DQR4SW5ANCNFSM4JFTVA7A> .

[sblionel]

Given that compiled Fortran objects are not interoperable with those from different compilers, much less modules, I don't see a way forward for this proposal. The standard offers features (especially submodules) that help library developers. Build from source works.

Keep in mind that the Fortran standard doesn't say anything about the world outside the "processor" (compiler). Source lines are delivered by fairies in the night, input and output files are up to the whim of the environment, etc. One can use any packaging system that suits your fancy. What does one do for C or C++?

[gronki]

Yes you are right about that. So the distribution system then should be based on source packages and compiled as the package is downloaded. Everyone: With that knowledge, what meta information would have to be included in the package? niedz., 3 lis 2019, 22:38 użytkownik Steve Lionel <notifications@github.com> napisał:

…

Given that compiled Fortran objects are not interoperable with those from different compilers, much less modules, I don't see a way forward for this proposal. The standard offers features (especially submodules) that help library developers. Build from source works. Keep in mind that the Fortran standard doesn't say anything about the world outside the "processor" (compiler). Source lines are delivered by fairies in the night, input and output files are up to the whim of the environment, etc. One can use any packaging system that suits your fancy. What does one do for C or C++? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#55?email_source=notifications&email_token=AC4NA3OVNFWL43TSF7J7XKDQR4ZGBA5CNFSM4JFTVA7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEC55RAQ#issuecomment-549181570>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC4NA3N4PEK6AOD5VJUP7P3QR4ZGBANCNFSM4JFTVA7A> .

[certik]

@sblionel, from the mentioned languages, Go and Rust are compiled, and Julia is a hybrid. Regarding your second question, there are two package managers specifically for C++: Conan and Vcpkg. A language neutral package managers that many recommend for C++ (and Fortran!) are Spack and Conda (already linked in the issue description above).

@gronki thanks for the feedback.

Fortran needs a standard way to create and distribute libraries. There is a lot to improve.

What is not clear at this stage is what things, if anything, needs to be improved in the Fortran standard itself. But there might be some things to improve there, and for that reason I would like to keep discussing it here.

I discussed this issue with many people, and there are generally two camps: a language specific package manager (like Julia, Python, ...) and those who advocate for packaging all languages (such as C++, Fortran, Python, ...) in language neutral package managers such as Conda or Spack.

I would much prefer if we can figure out ways to just use Conda, Spack or another solution, so that we do not need to maintain things ourselves. However, there might be some Fortran specific things that we might need to figure out.

Regarding building from source (Spack) or distributing binaries (Conda), I think we need both. We need to build from source, as that is what is needed on HPC to build an optimized static build for a specific architecture, but also be able to distribute binaries (Conda) is very helpful for users that just want to get something working quickly and do not want to wait hours to build all the dependencies.

[septcolor]

FWIW, D, Nim, Chapel (as well as Rust and Go) are also compiled languages, each of which has its package repository. Examples of major registries include...: Dub for D, Crates for Rust, and Gopm for Go, Hackage for Haskell, and so on.

Personally, I think this kind of package registry does not (necessarily) have to be "in the standard", but it is very nice if such a repository allows users to find "good" packages + install them efficiently with least troubles. The ideal situation is that a package registry (or manager) provides a search mechanism for candidate packages, show the degree of maintenance level explicitly (e.g. by showing validation/test results for major compilers/versions), show dependence (e.g. 3rd-party libraries/versions required), explicitly state license (to facilitate open-source use), and provide feedback mechanism such as popularity measures and issue reports...

[septcolor]

As for Rust, it is not only a new language but also the "most loved" one in the StackOverflow survey (2019)
https://insights.stackoverflow.com/survey/2019#most-loved-dreaded-and-wanted
and seems even considered as a possible replacement of C/C++
(according to Microsoft)
https://visualstudiomagazine.com/articles/2019/07/18/microsoft-eyes-rust.aspx
https://msrc-blog.microsoft.com/2019/07/18/we-need-a-safer-systems-programming-language/
so I guess it may also provide a useful reference for various aspects, including package management (in comparison to more traditional languages like C++ and Java).

[everythingfunctional]

In order for this to work well, you would need to tie all the packages together with a standardized build tool. I've started putting the beginnings of this together in my own packages, but I haven't formalized it or properly automated the package management side of it yet. Basically, I put together a build tool that can scan the source tree and determine the dependency tree. Then I just manually add the src folder to the list in the build system and use git submodules to manage the dependencies. Take a look here and let me know what you think.
It only works if everything is in modules and doesn't deal with submodules yet. I also have extended it to work with linking in C/C++ code in one project.

[traversaro]

Regarding your second question, there are two package managers specifically for C++: Conan and Vcpkg.

As it may be relevant that there have been some effort in the past to add support for Fortran in vcpkg, even if until now it has not been merged upstream:

• Fortran support microsoft/vcpkg#485
• Fortran support microsoft/vcpkg#1098
• Fortran Support microsoft/vcpkg#6939

[certik]

@traversaro thanks for the update!

We are developing a Fortran Package Manager (fpm) here: https://github.com/fortran-lang/fpm/, anyone is welcome to join us. It's very much work in progress, we will announce it once it is ready for users. If anyone wants to help us get there faster, please definitely join.

[everythingfunctional]

To follow on @certik comment, the latest developments have made fpm usable, provided you have no dependencies. That's the next step. I have some vague idea about how to implement a minimal version, but I need to find a few hours to dedicate to it.

[wolfv]

FPM is implemented in Haskell?

We are actively working on mamba (https://github.com/quantstack/mamba) again, which is becoming a complete rewrite of conda in C++ -- this will shed the dependency of conda for a Python interpreter and make it much more lightweight. In the end, you'll be able to drop a statically compiled binary on a system and use it as package manager -- and it works on Linux, Windows and OS X.
Mamba is also based upon well established dependency management libraries (libsolv, and libcurl / libarchive). So not too much NIH.

So far we're following conda's ideas very closely to make it 99% interoperable with existing conda packages and environments.

We are also toying around with the idea of adding source distribution capabilities, which would be part of mamba & the yet to be made mamba-build.

If the only thing you're missing from Conda is the ability to distribute source easily, maybe we can formulate a plan together to add this to mamba? I think yet-another language specific package manager is not the way to go (but I am not a fortran expert so there might be good reasons, which I didn't see in this thread at least).