What is a Julia standard library?

[fredrikekre]

What is a Julia standard library?

I have been thinking on this quite a lot recently, and wanted to gather some thoughts. I also feel like this is a question we have sidestepped a bit, and I think it would be nice to collect some ideas about the way forward for Julia's stdlib. In particular I think we should have a good definition of what a stdlib is, and what it means to be a stdlib. What follows are my thoughts on the situation, and I hope you all have some input, and hopefully we can formalize this a bit more.

Should stdlibs be regular packages?

While thinking about this I have come to the following conclusion; stdlibs would benefit a lot by trying to behave more like "regular" packages as much as possible. Why? It is a huge freedom to be a regular package. Packages in Julia work extremely well, and most of the modules in JuliaLang/julia/stdlib would be perfectly fine as packages. Here are some advantages of being a package (over the current JuliaLang/julia/stdlib style):

• Packages have separate repositories, meaning, among other things, that they have dedicated issue trackers such that issues don't get lost in the thousands of issues in JuliaLang/julia. It is also easier to get an overview of the package source code, since it is well contained.
• Packages are free to make new releases whenever they want, without caring about the release schedule of Julia. Most importantly this means you can get bug fix releases almost instantly (well, usually at least within a week!).
• Packages can set up their own docs, which means the focus of the documentation web page will be 100% on the package. As it is now, the stdlib docs are hidden deep within the Julia base manual (and TBH impossible to find).
• Packages have shorter CI times. A typical package does not take more than 10 minutes or so to run the test suite. This leads to faster development, since you don't have to wait for 2 hours to see the result for your PR.

With Pkg3 you will also have complete control over your environment, which IMO solves almost all problems with being a package.

Why do we need stdlibs?

So, then you may ask, why do we need something called a stdlib if packages are so great? Here are some reasons:

1. People like to use "official" packages and, therefore, stdlibs should be packages that are officially endorsed, supported and promoted by the JuliaLang organization. This gives credibility to the package and you should be able to trust that the package contains high quality code that is well tested and well documented.

Actually, that is the only reason I came up with, but it is an important one. What does it then mean to be officially supported by the organization?

• The Julia stdlibs are promoted, and linked to, from the base manual. We also list them on e.g. the homepage as the goto implementations and/or starting points for the different domains.
• Julia stdlibs are given extra consideration when changing things in the base language. I don't think we should necessarily run stdlib tests as part of base Julia CI, but perhaps keep an extra eye on the nightly builds. As with regular packages, I don't think you should expect stdlibs to work on Julia master at all times.
• Julia stdlibs have at least one active official maintainer.

Where should a stdlib live?

I think that JuliaLang/julia/stdlib is just a stopover for the modules that are there now. Eventually we need to move them out of the Julia repo in order to develop them more freely, and to simplify making new releases. We are already pushing the CI limit on the Julia repo, and moving them out would cut CI times by a lot (looking at you LinearAlgebra). In their own repos the stdlibs are better contained, and it will be easier to get an overview of the source. Developing a stdlib will also be much more pleasant when you don't have to build all of julia, and when you don't have to wait for 2 hours to hear back from CI.

Should we have a JuliaStdlib GitHub organization to gather the stdlibs? I don't think so. For example, there have been talks about making HTTP.jl a stdlib (whatever that means, trying to answer that in this post...). Should we then pull out HTTP.jl from the JuliaWeb organization? I think that would be a shame; HTTP.jl fits well in a JuliaWeb organization, and there is no reason to break that up. Likewise it would be a shame to break out StatsBase from JuliaStats.

There are some stdlibs that perhaps don't fit well in any organization (for example Test, Profile, DelimitedFiles etc.), and for those we can have a misc JuliaStdlib organization to host them.

After all, for end users, it does not really matter where the source is hosted and thus I don't think we have to gather them all. Users will read the manual, see that there is a stdlib called Test and then pkg> add Test without a worry in the world about where the source code lives.

How do you install a stdlib?

I don't think a stdlib needs to be installed by default -- it is very simple to pkg> add Package when you need it. Pkg3 makes installation much more pleasant, and pkg> add Package no longer grants you a coffee break, so having to add the stdlibs should not be an annoyance. One thing that is nice right now is that the stdlibs are included in the sysimg, which results in faster load times. At the moment this is needed for some of them, in particular LinearAlgebra and Pkg3 are pretty slow otherwise, but who cares if you can load the Test module instantly or not?

Hopefully we can work on decreasing the load times of packages in the future. We can also have a convenient way to add packages to the sysimg, and regenerate it. I initially thought that this option should only be for stdlibs, since they might not update as regularly as regular packages, but then, why shouldn't you be able to do that for any package?

The way forward

For Julia v0.7/v1.0 I think we should just leave the stdlibs in JuliaLang/julia/stdlib where they are, and state that they might not be included by default in v1.1, but they will still be supported and installable etc in future Julia versions. Then when v1.0 is out the window we simply move them all out to their own repos and reap the benefits.

Addendum

Furthermore, I think stdlibs should be allowed to be pirates.

TL;DR

stdlibs would benefit a lot from being more like regular packages and should live in their own repos.

[martinholters]

Note that not including packages in the sysimg not only increases loadtime (which we hopefully can work against in the future), but would also take away functionality that is currently present by automatically loading (without using) the package. E.g. the \ methods in LinearAlgebra. That would make such a change more breaking, not just pkg> add LinearAlgebra once, but using LinearAlgebra every time (to solve a linear system).

[StefanKarpinski]

I've largely been thinking along the same lines, @fredrikekre. Nice writeup. I was imagining that we would ship a stdlib of the "standard versions of standard libraries" but that would only serve as a depot that's always available if you want it and otherwise not be special at all. The biggest technical challenges are about overcoming load time issues and dependencies from Base to stdlibs like the one on \ that @martinholters brings up. One solution to the latter would be to automatically load LinearAlgebra from Base when someone uses \ and provide it that way.

[ViralBShah]

Much of what used to be available by default, like chol now requires using LinearAlgebra. The fact that \ works is perhaps odd, and probably should not, as it will create problems later on. It should receive the same treatment as the rest of linear algebra.

[JeffBezanson]

The plan all along (at least in my mind) has been for stdlibs to be as much like normal packages as possible. Past discussions have mentioned how, to a large extent, stdlib/ is just a staging area to make it easier to move code.

Even if the stdlibs are all moved to separate repos (which I agree has many advantages), there is still value in identifying a certain set of "recommended" or "default" packages. Installing a package is easy, but knowing which packages to install is not. And clearly, when you first build or install julia, you should not have to manually do 20 Pkg.adds. At the very least the package manager itself has to be available. You probably also want the REPL, since otherwise the first action is to run ./julia -e 'using Pkg; Pkg.add("REPL")' and then do the rest in the REPL. So at least some packages should be available by default, and that set is the stdlib.

Furthermore, I think stdlibs should be allowed to be pirates.

Agreed, but (as I constantly feel the need to point out) I'm not particularly against "type piracy" in any case.

[nalimilan]

I agree stdlib packages should be installed by default, even if they are regular packages.

Also, having the docs included in the Julia manual is useful. Even if it may not be very practical when you know exactly the name of the module/package you're looking for, it's good for users who don't know the details of where a feature lives. It allows the Julia manual to cover features that everybody expects to find in a modern language.

[ViralBShah]

I should point out that what is an stdlib package is a bit arbitrary and mostly by historical accident. One can easily imagine removing a few things and perhaps even adding some things.

[andreasnoack]

@martinholters I think the fact that \ still works without a deprecation is really just an artifact of the deprecation mechanism. My understanding was that when we decided to move linear algebra to the standard library, we decided that using LinearAlgebra was okay to require even when just solving a linear system of equations.

[JeffBezanson]

I agree with Viral --- while I think there should be some set of default packages, that doesn't mean the exact set of things in stdlib/ currently is perfect. Certainly some things could be removed and others added.

[Nosferican]

Would be great to have a customizable image based on optional combinations from a curated registry.

[ViralBShah]

Nothing stops you from building your own image today. People often do it with contrib/build_sysimg.jl.

[Nosferican]

Nice! Might wait for a blog post on how to do it though, but it is a great feature to have... Building the images for R at work was a hassle (i.e., and curated packages for Docker containers).

[asprionj]

Nothing stops you from building your own image today. People often do it with contrib/build_sysimg.jl.

build_sysimg.jl was deleted, see #25695. What's the best way of building a "reduced" custom system image, with only some selected stdlibs? create_sysimage in PackageCompiler.jl (see docs provides the filter_stdlibs flag, but it's not explained how it works and what it exactly does.

The desire for a smaller sysimage came up for embedded devices, see discourse thread. Not primarily because of the required storage, but because of the long load times (to read the sysimage from disk).