New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shrinking Base and Introducing a Standard Library #5155

Closed
tknopp opened this Issue Dec 15, 2013 · 81 comments

Comments

Projects
None yet
@tknopp
Contributor

tknopp commented Dec 15, 2013

In #4898 there is a discussion whether the reduced loading time of the wonderful static compilation work should be an argument to either shrink or expand the Base module.

As it seems to be difficult in general if things can go in to Base or go into a package (i.e. where to draw the line) I would like to propose the following:

  • Shrink Base so that the initial load time and memory consumption is even better
  • Introduce 4-5 modules forming the "standard library" of Julia.

Whether these standard modules should be automatically loaded is from my perspective only a minor discussion point. More important is that this behavior can be easily changed (e.g. in juliarc)

One concern might be that one cannot rely on what module has been loaded in the users environment. But this can be solved by simply always explicitly importing the standard libs when developing a package. "using StdLib" could be a shortcut to import all standard libraries.

It might make sense to offload the standard modules into packages to make the development flow easier. One could then pull in the standard lib when building Julia.

I have not a concrete proposal for standard modules but "LinAlg" and "Signal" (like scipy.signal) come to my mind.

@tknopp tknopp referenced this issue Dec 15, 2013

Merged

static compile round 2 #4898

11 of 11 tasks complete
@gitfoxi

This comment has been minimized.

Show comment
Hide comment
@gitfoxi

gitfoxi Dec 15, 2013

Contributor

To paraphrase myself from #4898, this seems like a case of premature optimization. No need to ape patterns from languages developed in the covered-wagon era of computing. If anything, look in the other direction. Assume a super-grid cloud cluster of overclocked n-way super-scalar VLIW stacked-die SMP GPUs.

Performance is quite a separate issue from namespace pollution. On that topic, more thought is required -- or possibly less. Is anyone complaining about the namespace yet? Might as well leave it alone until someone does.

Contributor

gitfoxi commented Dec 15, 2013

To paraphrase myself from #4898, this seems like a case of premature optimization. No need to ape patterns from languages developed in the covered-wagon era of computing. If anything, look in the other direction. Assume a super-grid cloud cluster of overclocked n-way super-scalar VLIW stacked-die SMP GPUs.

Performance is quite a separate issue from namespace pollution. On that topic, more thought is required -- or possibly less. Is anyone complaining about the namespace yet? Might as well leave it alone until someone does.

@JeffBezanson

This comment has been minimized.

Show comment
Hide comment
@JeffBezanson

JeffBezanson Dec 15, 2013

Member

@gitfoxi makes a good point --- being raised on OOP, we all reflexively see more modularity as a good thing, but we have to be clear what exactly we are trying to solve or obtain. It would be nice to have a "minimal julia" perhaps for constrained environments. But I think we can get these sorts of things without changing the default user experience.

see also #1906, #3128.

Member

JeffBezanson commented Dec 15, 2013

@gitfoxi makes a good point --- being raised on OOP, we all reflexively see more modularity as a good thing, but we have to be clear what exactly we are trying to solve or obtain. It would be nice to have a "minimal julia" perhaps for constrained environments. But I think we can get these sorts of things without changing the default user experience.

see also #1906, #3128.

@tknopp

This comment has been minimized.

Show comment
Hide comment
@tknopp

tknopp Dec 15, 2013

Contributor

I should have been clearer of the purpose of this proposal. Actually the main point is to have "privileged" packages (#1906) which are shipped with Julia. Then it might be easier to draw a clear line between what is in Base and what is in the Standard library. The "minimal Julia" mode would naturally evolve out of this reorganization but it is not the main driver for my proposal.

Maybe I was just surprised to see things like fft and sparse matrices in Base but no spatial filtering (gauss filter and so on)

If this proposal goes into the wrong direction, feel free to close it.

Contributor

tknopp commented Dec 15, 2013

I should have been clearer of the purpose of this proposal. Actually the main point is to have "privileged" packages (#1906) which are shipped with Julia. Then it might be easier to draw a clear line between what is in Base and what is in the Standard library. The "minimal Julia" mode would naturally evolve out of this reorganization but it is not the main driver for my proposal.

Maybe I was just surprised to see things like fft and sparse matrices in Base but no spatial filtering (gauss filter and so on)

If this proposal goes into the wrong direction, feel free to close it.

@StefanKarpinski

This comment has been minimized.

Show comment
Hide comment
@StefanKarpinski

StefanKarpinski Dec 15, 2013

Member

I don't actually think that doing using LinAlg at the top of a file before using linear algebra stuff is so bad. That's explicit and easy enough. My main objection to the way that "object-oriented" language like Python and Java do things is that they make the things you have to import so granular that every time you reach for new functionality you need to add imports. But we're not talking about anything like that. This would be one or two using lines at most.

Member

StefanKarpinski commented Dec 15, 2013

I don't actually think that doing using LinAlg at the top of a file before using linear algebra stuff is so bad. That's explicit and easy enough. My main objection to the way that "object-oriented" language like Python and Java do things is that they make the things you have to import so granular that every time you reach for new functionality you need to add imports. But we're not talking about anything like that. This would be one or two using lines at most.

@pao

This comment has been minimized.

Show comment
Hide comment
@pao

pao Dec 16, 2013

Member

It would be nice to have a "minimal julia" perhaps for constrained environments. But I think we can get these sorts of things without changing the default user experience.

As someone who gets to do embedded work, I think that's the right approach. When I know I'm doing a final deployment of a static codebase to a target system, give me the opportunity to strip that down, but don't make things (much) harder on the people who are actually writing the algorithms. The popularity of Simulink Coder/MATLAB Coder/Embedded Coder come from being able to have a full development environment on the one hand and deploy something much more basic on the other. Being able to do those things in a single language without a translation step is one of my hopes for Julia.

Member

pao commented Dec 16, 2013

It would be nice to have a "minimal julia" perhaps for constrained environments. But I think we can get these sorts of things without changing the default user experience.

As someone who gets to do embedded work, I think that's the right approach. When I know I'm doing a final deployment of a static codebase to a target system, give me the opportunity to strip that down, but don't make things (much) harder on the people who are actually writing the algorithms. The popularity of Simulink Coder/MATLAB Coder/Embedded Coder come from being able to have a full development environment on the one hand and deploy something much more basic on the other. Being able to do those things in a single language without a translation step is one of my hopes for Julia.

@tknopp

This comment has been minimized.

Show comment
Hide comment
@tknopp

tknopp Dec 16, 2013

Contributor

It seems that I am not good in expressing my thoughts. I was proposing that Base+Standard should have more functionality than currently in Base. And these could be loaded by default. This means that the "default user experience" is improved instead of degraded. And on top of this, for people doing embedding stuff, or simply people that want to program smartphone apps in Julia there would be an easy way to remove stuff that is not needed.

For me, Julia is not the better Matlab but actually one of the most promising alternative to programming in C++. Julia solves the "template issues" of C++ in such a beautiful way that it is hard to get back to C++ for the day work.

Contributor

tknopp commented Dec 16, 2013

It seems that I am not good in expressing my thoughts. I was proposing that Base+Standard should have more functionality than currently in Base. And these could be loaded by default. This means that the "default user experience" is improved instead of degraded. And on top of this, for people doing embedding stuff, or simply people that want to program smartphone apps in Julia there would be an easy way to remove stuff that is not needed.

For me, Julia is not the better Matlab but actually one of the most promising alternative to programming in C++. Julia solves the "template issues" of C++ in such a beautiful way that it is hard to get back to C++ for the day work.

@gitfoxi

This comment has been minimized.

Show comment
Hide comment
@gitfoxi

gitfoxi Dec 16, 2013

Contributor

If there's one thing we can all agree on, it's that C++ can go straight to hell.

Contributor

gitfoxi commented Dec 16, 2013

If there's one thing we can all agree on, it's that C++ can go straight to hell.

@StefanKarpinski

This comment has been minimized.

Show comment
Hide comment
@StefanKarpinski

StefanKarpinski Dec 16, 2013

Member

It's been a bit surprising but in retrospect makes sense that Julia is a pretty good C++ replacement. Both languages allow you more abstraction than C with similar performance, letting you consciously trade-off higher-level programming styles for efficiency, all the way down to writing C-like code and getting C-like performance (and frequently getting C-like performance without having to give up high-level style). Julia's parametric types and "everything is a template" approach has a very similar effect to C++ templates, but with far less brittleness and hassle. And of course, C++ uses operator overloading rampantly and Julia's operators, being simply generic functions with special syntax, give you operator overloading as it should be. So if the main things you liked about C++ were:

  1. performance
  2. generic programming
  3. operator and method overloading

then Julia is a phenomenal C++ replacement. If, on the other hand, the things you need from C++ extend to manual memory management, pointer arithmetic, and multithreading – which are very legitimate needs for some applications – then Julia is not such a great C++ replacement. But I think there are a lot of C++ programmers for whom the first three are really important and the last three are not.

Member

StefanKarpinski commented Dec 16, 2013

It's been a bit surprising but in retrospect makes sense that Julia is a pretty good C++ replacement. Both languages allow you more abstraction than C with similar performance, letting you consciously trade-off higher-level programming styles for efficiency, all the way down to writing C-like code and getting C-like performance (and frequently getting C-like performance without having to give up high-level style). Julia's parametric types and "everything is a template" approach has a very similar effect to C++ templates, but with far less brittleness and hassle. And of course, C++ uses operator overloading rampantly and Julia's operators, being simply generic functions with special syntax, give you operator overloading as it should be. So if the main things you liked about C++ were:

  1. performance
  2. generic programming
  3. operator and method overloading

then Julia is a phenomenal C++ replacement. If, on the other hand, the things you need from C++ extend to manual memory management, pointer arithmetic, and multithreading – which are very legitimate needs for some applications – then Julia is not such a great C++ replacement. But I think there are a lot of C++ programmers for whom the first three are really important and the last three are not.

@JeffBezanson

This comment has been minimized.

Show comment
Hide comment
@JeffBezanson

JeffBezanson Aug 12, 2014

Member

Very tentatively added to 0.4. This may not be as high-priority as other stuff though.

Member

JeffBezanson commented Aug 12, 2014

Very tentatively added to 0.4. This may not be as high-priority as other stuff though.

@timholy

This comment has been minimized.

Show comment
Hide comment
@timholy

timholy Aug 12, 2014

Member

If it is done after really good package precompilation, then it should be the second highest priority item for 0.4 (right after Keno's debugger) 😄

Member

timholy commented Aug 12, 2014

If it is done after really good package precompilation, then it should be the second highest priority item for 0.4 (right after Keno's debugger) 😄

@tknopp

This comment has been minimized.

Show comment
Hide comment
@tknopp

tknopp Aug 12, 2014

Contributor

@JeffBezanson Is there some general general platform (github issue or julia-dev mailing list) to discuss what should be targeted for 0.4?

@timholy Just wanted to add a comment on this but your message was faster :-)

My take: 0.4 has already various things that are "almost done" and if one targets 6 month for developing 0.4 I am not so sure if package precompilation and refactoring is doable. Esspecially as package precompilation can only be done by a limited number of people (Jameson, Keno, Jeff, ...).

In short: I am a big fan of package precompilation and the shrinking base. But please lets not add things to 0.4-projects which "may" happen. Better merge all the open important issues from the last half year, stabilize and release.

Contributor

tknopp commented Aug 12, 2014

@JeffBezanson Is there some general general platform (github issue or julia-dev mailing list) to discuss what should be targeted for 0.4?

@timholy Just wanted to add a comment on this but your message was faster :-)

My take: 0.4 has already various things that are "almost done" and if one targets 6 month for developing 0.4 I am not so sure if package precompilation and refactoring is doable. Esspecially as package precompilation can only be done by a limited number of people (Jameson, Keno, Jeff, ...).

In short: I am a big fan of package precompilation and the shrinking base. But please lets not add things to 0.4-projects which "may" happen. Better merge all the open important issues from the last half year, stabilize and release.

@timholy

This comment has been minimized.

Show comment
Hide comment
@timholy

timholy Aug 12, 2014

Member

@tknopp, look for issues with the 0.4-projects tag. Here's a link: https://github.com/JuliaLang/julia/milestones/0.4-projects

Member

timholy commented Aug 12, 2014

@tknopp, look for issues with the 0.4-projects tag. Here's a link: https://github.com/JuliaLang/julia/milestones/0.4-projects

@tknopp

This comment has been minimized.

Show comment
Hide comment
@tknopp

tknopp Aug 12, 2014

Contributor

Thanks Tim. Thats clear. But the question was more about whether there is a central place to discuss what should get the 0.4-projects tag.

Contributor

tknopp commented Aug 12, 2014

Thanks Tim. Thats clear. But the question was more about whether there is a central place to discuss what should get the 0.4-projects tag.

@ivarne

This comment has been minimized.

Show comment
Hide comment
@ivarne

ivarne Aug 12, 2014

Contributor

Isn't this issue mainly about moving things out of the Base module and into separate modules that can be imported separately with a using ... statement?

The job here seems to be that these modules must have explicit (and few) dependencies. Some sort of standard way of importing everything might also be good for interactive usage. Whether they are precompiled into the default sysimg, or lazy loaded from code or a dll is a totally different issue.

Contributor

ivarne commented Aug 12, 2014

Isn't this issue mainly about moving things out of the Base module and into separate modules that can be imported separately with a using ... statement?

The job here seems to be that these modules must have explicit (and few) dependencies. Some sort of standard way of importing everything might also be good for interactive usage. Whether they are precompiled into the default sysimg, or lazy loaded from code or a dll is a totally different issue.

@tknopp

This comment has been minimized.

Show comment
Hide comment
@tknopp

tknopp Aug 12, 2014

Contributor

@ivarne Yes exactly. Please have a look at my related comment #7926 (comment)

Technically it might make even more sense to put the modules into separate packages ("default packages") that are pulled when running make.

Contributor

tknopp commented Aug 12, 2014

@ivarne Yes exactly. Please have a look at my related comment #7926 (comment)

Technically it might make even more sense to put the modules into separate packages ("default packages") that are pulled when running make.

@timholy

This comment has been minimized.

Show comment
Hide comment
@timholy

timholy Aug 12, 2014

Member

Precompilation would be fine, but parse/compile from source code would be a massive step backwards. Some of us already have to wait ~1minute before Julia starts doing anything useful, and this would make it worse.

Member

timholy commented Aug 12, 2014

Precompilation would be fine, but parse/compile from source code would be a massive step backwards. Some of us already have to wait ~1minute before Julia starts doing anything useful, and this would make it worse.

@ivarne

This comment has been minimized.

Show comment
Hide comment
@ivarne

ivarne Aug 12, 2014

Contributor

This issue is not about making startup slower. It is about changing the structure of Base into less interdependent modules. All the new modules can (and should) be compiled with Julia as they do today.

Contributor

ivarne commented Aug 12, 2014

This issue is not about making startup slower. It is about changing the structure of Base into less interdependent modules. All the new modules can (and should) be compiled with Julia as they do today.

@tknopp

This comment has been minimized.

Show comment
Hide comment
@tknopp

tknopp Aug 12, 2014

Contributor

Yes although there might be different technical solutions. If we would have package precompilation it would be easier than without. One strategy would be:

Step 1: Move things into independent modules within base.
Step 2: Move the modules into individual packages that are automatically pulled during make (i.e. git submodules)
Step 3: Use precompiled packages.

One motivation of this proposal is that it hopefully lead to a much larger standard library. Image.jl is one examplary candidate. I can't live without gaussian multidimensional filtering in the Julia "standard library"...
More controversial might be to pull ploting and a GUI toolkit into the "standard library" but in my opinion this is would be a great way to move forward.

Contributor

tknopp commented Aug 12, 2014

Yes although there might be different technical solutions. If we would have package precompilation it would be easier than without. One strategy would be:

Step 1: Move things into independent modules within base.
Step 2: Move the modules into individual packages that are automatically pulled during make (i.e. git submodules)
Step 3: Use precompiled packages.

One motivation of this proposal is that it hopefully lead to a much larger standard library. Image.jl is one examplary candidate. I can't live without gaussian multidimensional filtering in the Julia "standard library"...
More controversial might be to pull ploting and a GUI toolkit into the "standard library" but in my opinion this is would be a great way to move forward.

@quinnj

This comment has been minimized.

Show comment
Hide comment
@quinnj

quinnj Aug 12, 2014

Member

I actually have a branch that allows a Make.user variable of NOBIGFLOAT that will exclude all BigFloat functionality. Some thoughts having gone through that process:

  • I think #6195 will help with this. The problem is that even with a minimal dependency like BigFloats that is mostly self-contained, there are little snippets that have sprouted up all over Base, which are quite awkward to just move into the MPFR module (constants.jl, printf.jl, etc.). The other option (which I've done in my branch) is to have a @BIGFLOAT macro that acts like @windows_only code.
  • Even with conditional modules, some dependencies like libuv and PRCE are literally everywhere. Those would take major surgery to remove or make optional.
  • That said, I think this is all worth the effort, though it would take a concerted effort.
Member

quinnj commented Aug 12, 2014

I actually have a branch that allows a Make.user variable of NOBIGFLOAT that will exclude all BigFloat functionality. Some thoughts having gone through that process:

  • I think #6195 will help with this. The problem is that even with a minimal dependency like BigFloats that is mostly self-contained, there are little snippets that have sprouted up all over Base, which are quite awkward to just move into the MPFR module (constants.jl, printf.jl, etc.). The other option (which I've done in my branch) is to have a @BIGFLOAT macro that acts like @windows_only code.
  • Even with conditional modules, some dependencies like libuv and PRCE are literally everywhere. Those would take major surgery to remove or make optional.
  • That said, I think this is all worth the effort, though it would take a concerted effort.
@StefanKarpinski

This comment has been minimized.

Show comment
Hide comment
@StefanKarpinski

StefanKarpinski Aug 12, 2014

Member

Moving functionality into packages and precompilation are completely orthogonal. We happen to create a system image before loading any external packages right now, but there's no reason that has to be the case. We can just as easily load Base, load various default packages, and then generate a system image.

Member

StefanKarpinski commented Aug 12, 2014

Moving functionality into packages and precompilation are completely orthogonal. We happen to create a system image before loading any external packages right now, but there's no reason that has to be the case. We can just as easily load Base, load various default packages, and then generate a system image.

@StefanKarpinski

This comment has been minimized.

Show comment
Hide comment
@StefanKarpinski

StefanKarpinski Aug 12, 2014

Member

Opened an issue just for precompilation: #7977.

Member

StefanKarpinski commented Aug 12, 2014

Opened an issue just for precompilation: #7977.

@timholy

This comment has been minimized.

Show comment
Hide comment
@timholy

timholy Sep 7, 2016

Member

I've thought about moving Profile out, and am happy to do the work esp. if it motivates others to begin on their pieces, but Profile requires a few profile-specific ccalls (platform-specific signal handling). Before we begin migration, it would be nice to have a strategy for providing pre-built binaries not just of (trimmed-down) Julia but also "common" packages.

#5155 (comment)

Member

timholy commented Sep 7, 2016

I've thought about moving Profile out, and am happy to do the work esp. if it motivates others to begin on their pieces, but Profile requires a few profile-specific ccalls (platform-specific signal handling). Before we begin migration, it would be nice to have a strategy for providing pre-built binaries not just of (trimmed-down) Julia but also "common" packages.

#5155 (comment)

@tkelman

This comment has been minimized.

Show comment
Hide comment
@tkelman

tkelman Sep 7, 2016

Contributor

Note that the migrated JuliaBox is now demonstrating some of the ways in which we're pre-bundling packages along with Julia. There have been some precompilation issues but we're working on fixing those.

Contributor

tkelman commented Sep 7, 2016

Note that the migrated JuliaBox is now demonstrating some of the ways in which we're pre-bundling packages along with Julia. There have been some precompilation issues but we're working on fixing those.

@tknopp

This comment has been minimized.

Show comment
Hide comment
@tknopp

tknopp Sep 7, 2016

Contributor

Question: Would it be required that the bundling happens during distribution of julia? What about doing this at the first execution of julia: I.e. ask the user: "do you want the minimal version or should we install some batteries?" Alternatively the minimal version could have a hint in the banner, what to execute so install the full version.

Contributor

tknopp commented Sep 7, 2016

Question: Would it be required that the bundling happens during distribution of julia? What about doing this at the first execution of julia: I.e. ask the user: "do you want the minimal version or should we install some batteries?" Alternatively the minimal version could have a hint in the banner, what to execute so install the full version.

@vchuravy

This comment has been minimized.

Show comment
Hide comment
@vchuravy

vchuravy Sep 7, 2016

Member

Also an important side note is that is would be nice to export the git history if possible to the package as well.

@andreasnoack and I have been discussing this for the sparse code.

Member

vchuravy commented Sep 7, 2016

Also an important side note is that is would be nice to export the git history if possible to the package as well.

@andreasnoack and I have been discussing this for the sparse code.

@simonbyrne

This comment has been minimized.

Show comment
Hide comment
@simonbyrne

simonbyrne Sep 7, 2016

Contributor

See JuliaMath/Primes.jl#12 for how it was done with Primes.jl

Contributor

simonbyrne commented Sep 7, 2016

See JuliaMath/Primes.jl#12 for how it was done with Primes.jl

@tkelman

This comment has been minimized.

Show comment
Hide comment
@tkelman

tkelman Sep 7, 2016

Contributor

Yes that's a good idea for code migrations, I did that with filter-branch for https://github.com/JuliaPackaging/Git.jl.

Contributor

tkelman commented Sep 7, 2016

Yes that's a good idea for code migrations, I did that with filter-branch for https://github.com/JuliaPackaging/Git.jl.

@vchuravy

This comment has been minimized.

Show comment
Hide comment
@tlnagy

This comment has been minimized.

Show comment
Hide comment
@tlnagy

tlnagy Sep 7, 2016

Contributor

Before we begin migration, it would be nice to have a strategy for providing pre-built binaries not just of (trimmed-down) Julia but also "common" packages.

I'm also worried about moving too many things out of Base before an effective "batteries-included" version of Julia is developed. Already I'm having to import a large number of packages before I'm able to get working, one of the things I liked least about python.

Would it be required that the bundling happens during distribution of julia? What about doing this at the first execution of julia

The latter would be much preferable. Having one meta-package that loads up base technical libraries at run time: Combinatorics, Primes, FFTW, QuadGK, DSP etc would be ideal. The user wouldn't see this after installing it (i.e. no using TechnicalJulia every time we fire up the repo).

Contributor

tlnagy commented Sep 7, 2016

Before we begin migration, it would be nice to have a strategy for providing pre-built binaries not just of (trimmed-down) Julia but also "common" packages.

I'm also worried about moving too many things out of Base before an effective "batteries-included" version of Julia is developed. Already I'm having to import a large number of packages before I'm able to get working, one of the things I liked least about python.

Would it be required that the bundling happens during distribution of julia? What about doing this at the first execution of julia

The latter would be much preferable. Having one meta-package that loads up base technical libraries at run time: Combinatorics, Primes, FFTW, QuadGK, DSP etc would be ideal. The user wouldn't see this after installing it (i.e. no using TechnicalJulia every time we fire up the repo).

@JeffBezanson

This comment has been minimized.

Show comment
Hide comment
@JeffBezanson

JeffBezanson Sep 7, 2017

Member

Next items that should be easy to move: SharedArrays and Poll (file watching).

Member

JeffBezanson commented Sep 7, 2017

Next items that should be easy to move: SharedArrays and Poll (file watching).

@JeffBezanson

This comment has been minimized.

Show comment
Hide comment
@JeffBezanson

JeffBezanson Sep 20, 2017

Member

Closing in favor of #18795 and more specific issues for pieces to remove.

Member

JeffBezanson commented Sep 20, 2017

Closing in favor of #18795 and more specific issues for pieces to remove.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment