Shrinking Base and Introducing a Standard Library

[tknopp]

In #4898 there is a discussion whether the reduced loading time of the wonderful static compilation work should be an argument to either shrink or expand the Base module.

As it seems to be difficult in general if things can go in to Base or go into a package (i.e. where to draw the line) I would like to propose the following:

• Shrink Base so that the initial load time and memory consumption is even better
• Introduce 4-5 modules forming the "standard library" of Julia.

Whether these standard modules should be automatically loaded is from my perspective only a minor discussion point. More important is that this behavior can be easily changed (e.g. in juliarc)

One concern might be that one cannot rely on what module has been loaded in the users environment. But this can be solved by simply always explicitly importing the standard libs when developing a package. "using StdLib" could be a shortcut to import all standard libraries.

It might make sense to offload the standard modules into packages to make the development flow easier. One could then pull in the standard lib when building Julia.

I have not a concrete proposal for standard modules but "LinAlg" and "Signal" (like scipy.signal) come to my mind.

[gitfoxi]

To paraphrase myself from #4898, this seems like a case of premature optimization. No need to ape patterns from languages developed in the covered-wagon era of computing. If anything, look in the other direction. Assume a super-grid cloud cluster of overclocked n-way super-scalar VLIW stacked-die SMP GPUs.

Performance is quite a separate issue from namespace pollution. On that topic, more thought is required -- or possibly less. Is anyone complaining about the namespace yet? Might as well leave it alone until someone does.

[JeffBezanson]

@gitfoxi makes a good point --- being raised on OOP, we all reflexively see more modularity as a good thing, but we have to be clear what exactly we are trying to solve or obtain. It would be nice to have a "minimal julia" perhaps for constrained environments. But I think we can get these sorts of things without changing the default user experience.

see also #1906, #3128.

[tknopp]

I should have been clearer of the purpose of this proposal. Actually the main point is to have "privileged" packages (#1906) which are shipped with Julia. Then it might be easier to draw a clear line between what is in Base and what is in the Standard library. The "minimal Julia" mode would naturally evolve out of this reorganization but it is not the main driver for my proposal.

Maybe I was just surprised to see things like fft and sparse matrices in Base but no spatial filtering (gauss filter and so on)

If this proposal goes into the wrong direction, feel free to close it.

[StefanKarpinski]

I don't actually think that doing using LinAlg at the top of a file before using linear algebra stuff is so bad. That's explicit and easy enough. My main objection to the way that "object-oriented" language like Python and Java do things is that they make the things you have to import so granular that every time you reach for new functionality you need to add imports. But we're not talking about anything like that. This would be one or two using lines at most.

[pao]

It would be nice to have a "minimal julia" perhaps for constrained environments. But I think we can get these sorts of things without changing the default user experience.

As someone who gets to do embedded work, I think that's the right approach. When I know I'm doing a final deployment of a static codebase to a target system, give me the opportunity to strip that down, but don't make things (much) harder on the people who are actually writing the algorithms. The popularity of Simulink Coder/MATLAB Coder/Embedded Coder come from being able to have a full development environment on the one hand and deploy something much more basic on the other. Being able to do those things in a single language without a translation step is one of my hopes for Julia.

[tknopp]

It seems that I am not good in expressing my thoughts. I was proposing that Base+Standard should have more functionality than currently in Base. And these could be loaded by default. This means that the "default user experience" is improved instead of degraded. And on top of this, for people doing embedding stuff, or simply people that want to program smartphone apps in Julia there would be an easy way to remove stuff that is not needed.

For me, Julia is not the better Matlab but actually one of the most promising alternative to programming in C++. Julia solves the "template issues" of C++ in such a beautiful way that it is hard to get back to C++ for the day work.

[gitfoxi]

If there's one thing we can all agree on, it's that C++ can go straight to hell.

[StefanKarpinski]

It's been a bit surprising but in retrospect makes sense that Julia is a pretty good C++ replacement. Both languages allow you more abstraction than C with similar performance, letting you consciously trade-off higher-level programming styles for efficiency, all the way down to writing C-like code and getting C-like performance (and frequently getting C-like performance without having to give up high-level style). Julia's parametric types and "everything is a template" approach has a very similar effect to C++ templates, but with far less brittleness and hassle. And of course, C++ uses operator overloading rampantly and Julia's operators, being simply generic functions with special syntax, give you operator overloading as it should be. So if the main things you liked about C++ were:

1. performance
2. generic programming
3. operator and method overloading

then Julia is a phenomenal C++ replacement. If, on the other hand, the things you need from C++ extend to manual memory management, pointer arithmetic, and multithreading – which are very legitimate needs for some applications – then Julia is not such a great C++ replacement. But I think there are a lot of C++ programmers for whom the first three are really important and the last three are not.

[JeffBezanson]

Very tentatively added to 0.4. This may not be as high-priority as other stuff though.

[timholy]

If it is done after really good package precompilation, then it should be the second highest priority item for 0.4 (right after Keno's debugger) 😄

[tknopp]

@JeffBezanson Is there some general general platform (github issue or julia-dev mailing list) to discuss what should be targeted for 0.4?

@timholy Just wanted to add a comment on this but your message was faster :-)

My take: 0.4 has already various things that are "almost done" and if one targets 6 month for developing 0.4 I am not so sure if package precompilation and refactoring is doable. Esspecially as package precompilation can only be done by a limited number of people (Jameson, Keno, Jeff, ...).

In short: I am a big fan of package precompilation and the shrinking base. But please lets not add things to 0.4-projects which "may" happen. Better merge all the open important issues from the last half year, stabilize and release.

[timholy]

@tknopp, look for issues with the 0.4-projects tag. Here's a link: https://github.com/JuliaLang/julia/milestones/0.4-projects

[tknopp]

Thanks Tim. Thats clear. But the question was more about whether there is a central place to discuss what should get the 0.4-projects tag.

[ivarne]

Isn't this issue mainly about moving things out of the Base module and into separate modules that can be imported separately with a using ... statement?

The job here seems to be that these modules must have explicit (and few) dependencies. Some sort of standard way of importing everything might also be good for interactive usage. Whether they are precompiled into the default sysimg, or lazy loaded from code or a dll is a totally different issue.

[tknopp]

@ivarne Yes exactly. Please have a look at my related comment #7926 (comment)

Technically it might make even more sense to put the modules into separate packages ("default packages") that are pulled when running make.

[timholy]

Precompilation would be fine, but parse/compile from source code would be a massive step backwards. Some of us already have to wait ~1minute before Julia starts doing anything useful, and this would make it worse.

[ivarne]

This issue is not about making startup slower. It is about changing the structure of Base into less interdependent modules. All the new modules can (and should) be compiled with Julia as they do today.

[tknopp]

Yes although there might be different technical solutions. If we would have package precompilation it would be easier than without. One strategy would be:

Step 1: Move things into independent modules within base.
Step 2: Move the modules into individual packages that are automatically pulled during make (i.e. git submodules)
Step 3: Use precompiled packages.

One motivation of this proposal is that it hopefully lead to a much larger standard library. Image.jl is one examplary candidate. I can't live without gaussian multidimensional filtering in the Julia "standard library"...
More controversial might be to pull ploting and a GUI toolkit into the "standard library" but in my opinion this is would be a great way to move forward.

[quinnj]

I actually have a branch that allows a Make.user variable of NOBIGFLOAT that will exclude all BigFloat functionality. Some thoughts having gone through that process:

• I think julep: conditional modules #6195 will help with this. The problem is that even with a minimal dependency like BigFloats that is mostly self-contained, there are little snippets that have sprouted up all over Base, which are quite awkward to just move into the MPFR module (constants.jl, printf.jl, etc.). The other option (which I've done in my branch) is to have a @BIGFLOAT macro that acts like @windows_only code.
• Even with conditional modules, some dependencies like libuv and PRCE are literally everywhere. Those would take major surgery to remove or make optional.
• That said, I think this is all worth the effort, though it would take a concerted effort.

[StefanKarpinski]

Moving functionality into packages and precompilation are completely orthogonal. We happen to create a system image before loading any external packages right now, but there's no reason that has to be the case. We can just as easily load Base, load various default packages, and then generate a system image.

[StefanKarpinski]

Opened an issue just for precompilation: #7977.

[timholy]

I've thought about moving Profile out, and am happy to do the work esp. if it motivates others to begin on their pieces, but Profile requires a few profile-specific ccalls (platform-specific signal handling). Before we begin migration, it would be nice to have a strategy for providing pre-built binaries not just of (trimmed-down) Julia but also "common" packages.

#5155 (comment)

[tkelman]

Note that the migrated JuliaBox is now demonstrating some of the ways in which we're pre-bundling packages along with Julia. There have been some precompilation issues but we're working on fixing those.

[tknopp]

Question: Would it be required that the bundling happens during distribution of julia? What about doing this at the first execution of julia: I.e. ask the user: "do you want the minimal version or should we install some batteries?" Alternatively the minimal version could have a hint in the banner, what to execute so install the full version.

[vchuravy]

Also an important side note is that is would be nice to export the git history if possible to the package as well.

@andreasnoack and I have been discussing this for the sparse code.

[simonbyrne]

See JuliaMath/Primes.jl#12 for how it was done with Primes.jl

[tkelman]

Yes that's a good idea for code migrations, I did that with filter-branch for https://github.com/JuliaPackaging/Git.jl.

[vchuravy]

For sparse I am using https://gist.github.com/vchuravy/81de73d34ed2b24de3123f50bb3c5bbd with the results at https://github.com/vchuravy/ExperimentalSparse4

[tlnagy]

Before we begin migration, it would be nice to have a strategy for providing pre-built binaries not just of (trimmed-down) Julia but also "common" packages.

I'm also worried about moving too many things out of Base before an effective "batteries-included" version of Julia is developed. Already I'm having to import a large number of packages before I'm able to get working, one of the things I liked least about python.

Would it be required that the bundling happens during distribution of julia? What about doing this at the first execution of julia

The latter would be much preferable. Having one meta-package that loads up base technical libraries at run time: Combinatorics, Primes, FFTW, QuadGK, DSP etc would be ideal. The user wouldn't see this after installing it (i.e. no using TechnicalJulia every time we fire up the repo).

[JeffBezanson]

Next items that should be easy to move: SharedArrays and Poll (file watching).

[JeffBezanson]

Closing in favor of #18795 and more specific issues for pieces to remove.