Support building of system image in binary builds

[staticfloat]

This commit adds a few new pieces of functionality:

• The contrib/build_sysimg.jl script which builds a new system image. This method can save the system image wherever the user desires, e.g. it could be stored in ~/.julia, to allow for per-user system images each customized with packages in their own userimg.jl, for instance. Or on a restricted system, this allows for creation of a system image without root access.
• The removal of compile-time JULIA_CPU_TARGET, in favor of runtime --cpu-target/-C command-line flags which default to "native" but can be set to "native", "i386" or "core2". This allows the creation of a system image targeting user-supplied cpu features, e.g. cd base; ../julia -C i386 --build /tmp/sys_i386 sysimg.jl.
• I implemented runtime selection of the cpu target by adding a new member to the jl_compileropts_t structure called cpu_target.
• Because all julia executables are now created equal, (rather than before where a julia executable needed to have the same JULIA_CPU_TARGET set internally as the system image had when it was built) we need to know what CPU feature set the system image is targeting before we initialize code generation. So a new function jl_get_system_image_cpu_target() is exported, which does exactly what it sounds like.
• I added newlines to the end of a few error messages.
• I found an old parser option -T which hadn't been removed yet, so I took the opportunity to do so.

When testing this change out, I found this gist helpful to put into my ~/.juliarc.jl

[staticfloat]

Some new results now that this PR has been re-created to run on top of the latest master. As last time, if I create three separate system images, each targeting a different cpu architecture:

include("contrib/build_sysimg.jl")
f(z) = build_sysimg("/tmp/sys_$z", cpu_target=z, force=true)
f("native"); f("core2"); f("i386")

And then run test/perf/micro/perf.jl with these different system images, I get the following performance:

$ for sysimg in native core2 i386; do time ../../../julia -J /tmp/sys_$sysimg.ji perf.jl; done
---------------------------------------------
cpu_target: native
julia,fib,0.070708,0.075016,0.071697,0.001870
julia,parse_int,0.231378,0.242547,0.237773,0.004061
julia,mandel,0.231137,0.231905,0.231443,0.000322
julia,quicksort,0.467930,0.483488,0.474949,0.006492
julia,pi_sum,46.400207,49.820934,47.782845,1.350097
julia,rand_mat_stat,21.155895,38.054953,28.303507,8.004958
julia,rand_mat_mul,61.888229,206.132585,108.874366,57.685708
julia,printfd,22.861897,24.619546,23.762590,0.825247

real    0m3.521s
user    0m4.165s
sys     0m0.552s
---------------------------------------------
cpu_target: core2
julia,fib,0.068215,0.068890,0.068367,0.000293
julia,parse_int,0.204745,0.218237,0.209300,0.005236
julia,mandel,0.227594,0.228392,0.227832,0.000331
julia,quicksort,0.407911,0.419301,0.413773,0.005027
julia,pi_sum,24.130807,26.410157,24.669580,0.978664
julia,rand_mat_stat,17.719686,39.570009,27.537551,9.635108
julia,rand_mat_mul,41.164079,70.174454,57.852161,11.354887
julia,printfd,22.640426,23.161379,22.953174,0.232171

real    0m3.328s
user    0m3.775s
sys     0m0.527s
---------------------------------------------
cpu_target: i386
julia,fib,0.068241,0.069031,0.068417,0.000344
julia,parse_int,0.205714,0.230857,0.216461,0.009434
julia,mandel,0.343922,0.371608,0.356498,0.012273
julia,quicksort,0.570250,0.662391,0.606683,0.035686
julia,pi_sum,24.130515,26.616021,24.763802,1.054345
julia,rand_mat_stat,25.283556,83.628569,50.180926,25.312187
julia,rand_mat_mul,45.546880,66.677871,53.709918,7.823724
julia,printfd,22.386678,24.675346,23.604397,0.875454

real    0m3.270s
user    0m3.744s
sys     0m0.579s

Note that the cpu_target: printing is coming from my ~/.juliarc.jl which is similar to this gist.

[staticfloat]

The above numbers seem to indicate that there is indeed a difference in the system images now, so that's great. There must have been something wrong with LLVM and my Haswell CPU last time I tried this, (which is indeed what Keno guessed at in the previous PR). Particularly, we can see an increase in performance in quicksort, mandel, rand_mat_stat, etc. as we move from i386 to core2 to native.

[Keno]

Why is native slower than core2?

[staticfloat]

I noticed that native seems to have trouble on a few tests. I don't know why it's slower. The only thing I can guess at is that we're doing something funky with AVX instructions, since running analyze-x86 on the two system images shows:

$ analyze-x86 sys_core2.dylib 
ainstructions:
 cpuid: 0        nop: 948        call: 0         count: 615999
 mmx:    334528
 sse:    11627
 sse2:   945
 sse3:   12
 ssse3:  4
$ analyze-x86 sys_native.dylib 
instructions:
 cpuid: 0        nop: 918        call: 0         count: 616248
 mmx:    334156
 avx:    13367

So you can see that all the SSE* instructions in the core2 build are getting moved to AVX, plus a LOT more besides, but the overall count remains almost the same, increasing a bit on the native side. It sounds to me like there's some kind of inefficiency going on when we switch over to AVX, but I can't pretend to know what I'm talking about. ;)

[nalimilan]

@staticfloat So what's the replacement for make JULIA_CPU_TARGET=core2 now?

[staticfloat]

First off, something I didn't know until recently is that you should be using MARCH=core2 in your binary builds now. This will enforce the cpu instructions for both julia, libjulia, and sys.so to be the same, which is nice. JULIA_CPU_TARGET in the makefile will still work. You can see that it is getting passed to the system image build commandline via the -C option.

[nalimilan]

@staticfloat Thanks. Ideally I would like Julia to run on any i386,I'm only passing JULIA_CPU_TARGET=core2 because with i386 the tests fail at the moment (#7185). So I'd rather not pass MARCH=core2. But I'll pass MARCH=i386 when it gets fixed.

[vtjnash]

I forgot this wasn't merged yet, probably because it's hard to get eyes on an issue that helps binary users, and not source-builds.

@JeffBezanson any concerns with this?

[staticfloat]

I'm currently rebasing, but shouldn't need any large changes, I don't think.

[tkelman]

In its current form, your find_system_linker() isn't going to work for anyone on Windows. We should turn off the sys.dll linking step on Windows until upgrading LLVM anyway. After that happens, we can revisit. Easiest approach is WinRPM.install("binutils") to download ld, but then it's going to live under the WinRPM deps directory - not on the path, but easy enough to figure out.

More importantly, will this be installed somewhere in a predictable location? Or do we tell people to download it off github and just include it?

[staticfloat]

I'm thinking we could shove it into share/julia somewhere. It seems like it'll be pretty useful.

I tried WinRPM.install("binutils") and it didn't install anything, it gave me an error saying that it didn't know what binutils was. Do I need to do anything other than:

Pkg.add("WinRPM")
using WinRPM
WinRPM.update()
WinRPM.install("binutils")

[tkelman]

The opensuse build service is going through another broken phase, I wasn't able to install Nettle yesterday for example (bad timing, as I was trying to get a colleague to try Julia and installing IJulia couldn't work without WinRPM). I'm pretty sure I have gotten WinRPM.install("binutils") to work before, the package is at https://build.opensuse.org/package/show/windows:mingw:win64/mingw64-binutils so it should work when it's not feeling temperamental. We really need to find a way to isolate ourselves from that breakage somehow, my best idea on the mailing list was figuring out how to mirror the RPM repository ourselves and run a periodic sanity check (on one of your buildbots if you can get Windows working?) before pulling in updates from upstream.

[staticfloat]

Yeah, sounds good. Let's add it to the todo list. If you can find/write a quick tutorial on setting up an RPM repository, we can host it on AWS.

[tkelman]

@nalimilan do you know by any chance? Otherwise I'll go googling, something else to learn how to do.

[tkelman]

@staticfloat try installing binutils again, I was able to get it to work again (from Windows - I'm not sure whether WinRPM is supposed to work elsewhere). curl is still broken though, it's unresolvable because one of its dependencies has an issue. I have a request in to fix it (https://build.opensuse.org/request/show/258347), but Tk is also broken and that one I don't know how to fix.

[staticfloat]

Alright, I've attempted some Windows compatibility, but I don't have any from-source windows builds, so I can't test this branches changes on windows. Could someone else try it out? This works pretty well on mac for me.

Specifically, testing out the userimg_path option would be nice, and of course, the actual linking step.

[staticfloat]

Updated to not use which, to cleanup properly if userimg_path is passed, and to actually run if userimg_path is not passed.

[tanmaykm]

This will be great to have. Packages included through userimg.jl load much faster.

$ time julia -e 'using Gadfly'
real    0m48.120s
user    0m48.326s
sys 0m1.716s

$ time julia -J /home/tanmaykm/userimg/sys.ji -e 'using Gadfly'
real    0m4.565s
user    0m4.842s
sys 0m1.013s

But I noticed a slowdown in startup time. Is this to be expected?

$ time julia -e ''
real    0m0.242s
user    0m0.255s
sys 0m0.205s

$ time julia -J /home/tanmaykm/userimg/sys.ji -e ''
real    0m4.259s
user    0m4.805s
sys 0m1.416s

[staticfloat]

It's possible that is simply due to the fact that your system image now includes Gadfly and all dependencies. If you build one without Gadfly included at all, do you see a similar slowdown?

[tkelman]

Oh right, distributions probably won't like putting a .jl file next to /usr/bin/julia, will they @nalimilan. It would be nice to make this not-a-pain to type. We have relative-path DATAROOTDIR now which might be more appropriate?

[nalimilan]

I'm not sure what the distributions' policy is with regard to files with extensions in /usr/bin, but there are very few files named like that in that directory, and I don't find typing julia-sysimg.jl to run it very nice. Maybe put the script in DATADIR, and have an argument to the julia executable to run it? This also makes it easier to discover the feature, just by using julia --help or man julia.

[tkelman]

I'm not sure if we expect people to run this often enough for it to be worth a command-line flag (also command-line flags aren't very Windows-friendly, since the default command prompt is terrible and people don't use it much). I personally wouldn't mind having this available in base like Base.runtests, but I think @JeffBezanson vetoed that in some other issue.

[staticfloat]

@JeffBezanson how would you best like this code to be callable? The comment in question is this one, and I agree with you that the code doesn't need to be in the standard library. The way I see it, we have three options:

• Install this code into usr/bin as something like build_sysimg.jl. This would allow everyone to access it via their PATH. Not that people should be doing this as part of their morning routine, but making it easy to use is a good thing.
• Install this code into usr/share/julia as build_sysimg.jl, but create a Base.build_sysimg() that subs out to build_sysimg.jl. This is my least favorite option, as at that point, why not just put it in Base?!
• Install this code into usr/share/julia and just require users to go and find it. It should, of course, be clearly documented as to how to find it and run it. This is my favorite option, as I really do think this is a "once per install" kind of operation that users are going to do.

[tkelman]

I guess include(joinpath(JULIA_HOME, Base.DATAROOTDIR, "julia", "build_sysimg.jl")) isn't too bad for option 3. Remember distributions will be customizing DATAROOTDIR so it'd be best if the same doc instructions applied there as well.

[staticfloat]

Yeah, on second thought, I definitely like option 3 the best, and I can't imagine Jeff has anything to say against it. I'm trying to find a good place to put documentation about this. Can anyone give me a hint as to a good place to advertise this functionality?

[tkelman]

My vote would be either for http://docs.julialang.org/en/latest/devdocs/julia/ or a new section in devdocs about system image building / modifying code in base.

[staticfloat]

Ah, devdocs! Brilliant. Also, glad to see someone else is up late alongside me, Tony. Unless, of course, you are not in fact int the timezone I think you are.

[tkelman]

What are these "time zones" you speak of? Aside from a spike around 2 am, according to https://osrc.dfm.io/tkelman/#schedule my likelihood of being online is almost uniform over the day.

[staticfloat]

I'm going to merge this tonight, barring any discussions.

[vchuravy]

Maybe squash the commits a bit before merging?

[staticfloat]

Squashed and rebased. This is by far the most rebased PR I have ever worked on. Once that little travis light turns green, I'm going to merge.

[staticfloat]

We've still got windows breakage, but that doesn't seem to be a part of this. MERGING.

[tkelman]

We've still got windows breakage, but that doesn't seem to be a part of this.

Yeah, not your fault. That was premature merging of Jameson's stuff before it actually worked on Windows.

[tkelman]

@staticfloat this will require manual conflict resolution to backport, can you prepare a PR while we test this on master?

edit: backport is at #9376