Upgrade to LLVM 3.9

[StefanKarpinski]

Related but closed: #18863. What's the status of this?

[vchuravy]

#19054 had the important bits and pieces of #18863 and from what I can tell we need:

• Patch llvm-3.8.0_threads for 3.9 Add Windows patches for LLVM 3.9 #19138
• Patch llvm-win64-reloc-dwarf for 3.9 (also Add Windows patches for LLVM 3.9 #19138 but may need double-checking)
• CMake >= 3.4.3 on Travis (done in RFC: Change LLVM version to 3.9.1 #19678)
• Update buildbots (see Upgrade cmake on all buildbots JuliaCI/julia-buildbot#54)
• Homebrew tap (see merge bottled llvm39-julia to master staticfloat/homebrew-julia#229)

[tkelman]

• make sure tests pass everywhere (seems obvious, but last I checked 3.8 they didn't - see backtraces in Add Windows patches for LLVM 3.9 #19138, neither win32 nor win64 can bootstrap successfully with 3.9 yet) fixed by Disable szclass 12 for i686 systems. #19647
• update appveyor binaries (see https://github.com/JuliaLang/julia/pull/14623/files for what this looked like last time - done in RFC: Change LLVM version to 3.9.1 #19678)
• add a script in contrib to make it easier for people on Linux to get a new cmake if their distro doesn't have one (nice-to-have, I'll write this - Add a script in contrib to download a cmake binary #19632)

[tkelman]

Note that the threads patch was initially borrowed from rust (who have the same toolchain C++11 threads issue we do with mingw), so we should check if they have a rebased version we could use to save ourselves a bit of time there. (edit: they do)

[vchuravy]

Something else I just noticed is that the Intel Compiler 2016_update2 is not able to build llvm. See https://sft.its.cern.ch/jira/browse/ROOT-8233 and https://software.intel.com/en-us/forums/intel-c-compiler/topic/640702

[tkelman]

Looks like that's icc's bug, but llvm continues to do an incredible job of finding new ways to break every other compiler...

[ihnorton]

Note that the threads patch was initially borrowed from rust (who have the same toolchain C++11 threads issue we do with mingw)

Worth keeping an eye on: https://github.com/lhmouse/mcfgthread

There was some discussion on the gcc list in April, but I don't see any response or recent activity, so it is unclear if/when it might go in to mainline. There is some further discussion/development on mingw-w64-public.

lhmouse does have a forked toolchain build here.

(related: https://github.com/meganz/mingw-std-threads -- header-only, but does not implement call_once std::async, and a handful of other things. call_once was the only issue I encountered when I tried building LLVM against it a while back)

[tkelman]

Yes, though unless that somehow gets backported to gcc 4, we'll need to worry about https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77333... or come up with a proper fix to gmp's build system so it works with clang on windows and start using that (ref #15488 which doesn't work in cross-compile).

[Keno]

Another issue we'll start seeing is that LLVM symbols conflict with graphics drivers on Linux distributions that have (stupidly) chosen to dynamically link LLVM into their graphics drivers. I have a patch (https://reviews.llvm.org/D13330) that adds capabilities to clang, to avoid generating the problematic symbols, but I doubt similar capabilities will make it into gcc anytime soon.

[kpamnany]

Does this work at all on Linux? Need LLVM 3.9 to start testing Julia performance on KNL.

[yuyichao]

Stock/patched LLVM 3.9.0 should work with no problem on x86/x64 linux.

[andreasnoack]

@kpamnany I had to use LLVM-svn on KNL but it worked fine (~a month ago.)

[yuyichao]

@kpamnany Apply https://reviews.llvm.org/D26123 or a commit before https://reviews.llvm.org/rL285426 if you want a llvm-svn build without assertion enabled.

[StefanKarpinski]

Bump. What's the status here?

[tkelman]

#19138 can be merged, but bootstrap crashes on both win32 and win64 last time I tried with 3.9.

[StefanKarpinski]

What's necessary to get this not to crash?

[ViralBShah]

@Keno is necessary, perhaps?

[vchuravy]

With #19138 both Win32 and Win64 build successfully and with #19493 my local Win64 builds passes all tests.

I didn't have much time to look into the Win32 build, but I am seeing a failure in the threading tests for a = Atomic{Int64}(); a[] = 1.

[tkelman]

@vchuravy that's what https://reviews.llvm.org/D27653 is intended to address, right?

[yuyichao]

Yes. But I don't think that's the right solution. I think the long term solution is to make sure we can find libatomic on windows (which can possibly be MSVC compatible once MSVC supports it and if gcc devs do the right thing) and the simplest short term solution is to remove szclass 12 on 32bit.

[tkelman]

remove szclass 12 on 32bit

What effect would that have? Things are working without requiring libatomic on llvn 3.7. There actually is no toolchain option that can build llvm 3.7 (and the rest of release-0.5 julia) correctly on win32 and includes libatomic as far as I'm aware.

[yuyichao]

What effect would that have?

Larger memory consumption.

Things are working without requiring libatomic on llvn 3.7

Due to a LLVM bug.

[tkelman]

What other consequences does the llvm bug have? A memory regression on 32 bit might be fine if it's not too large. Can we avoid libatomic or do we need to fix gcc 5+ (or libgmp's build system) to be able to have a toolchain that can build both llvm 3.9 and release-0.5 julia?

[yuyichao]

What other consequences does the llvm bug have?

The instructions it generates violates the platform ABI, which shouldn't cause too much issue due to our limited use of it currently.

Can we avoid libatomic

That's why I mentioned a short term solution? Or are you talking about something else?

[tkelman]

crossing threads in my head here, sorry. what needs patching to remove szclass 12?

[yuyichao]

1857f07 apart from the MAX_ALIGN part. It's quite easy to do if we decide that we need a short term solution (i.e. if we cannot get libatomic to work before we want to use llvm39 on windows).

[tkelman]

So modifying the ifdef conditionals there to also include win32? What about i686 linux? Not sure whether llvm 3.9 has been tested there yet but we have buildbots that run tests there.

I can test this if anyone can propose the right patch; so could anyone else who has a win32 source build handy but I'm guessing there aren't that many aside from me.

[yuyichao]

So modifying the ifdef conditionals there to also include win32

Yes.

What about i686 linux

I suspect this is less useful than win32 so may as well include it as well if we are going to change it on win32.

Not sure whether llvm 3.9 has been tested there yet

I've tested it and the tests should all pass.

I can test this if anyone can propose the right patch

If we decide to do this I'll write the patch. Or maybe we can test it first anyway?

[vchuravy]

From my point of view removing szclass 12 on 32bit (both Linux and Windows) is the right thing to do for now and once we can upgrade GCC we can revisit this.

I can test this over the weekend.

[vchuravy]

#19647 implements @yuyichao suggestion and I tested it on 32bit linux and windows (via wine)

[tkelman]

Tests pass for me with 3.9.0. We're in that annoying period where LLVM has tagged 3.9.1 but not released source tarballs, so maybe we should make our own again temporarily (if someone does this, may as well try to be consistent with how llvm makes their source releases look, in terms of a .src or whatever naming convention they use inside the tarball).

Someone will need to submit an llvm39-julia homebrew formula to the homebrew-julia tap (should leave the llvm37-julia alone as it is, best not to delete it), may as well make it 3.9.1 plus all our patches for consistency with how source builds will look.

[ViralBShah]

Noting the homebrew tap for llvm39 PR here: staticfloat/homebrew-julia#229

@vchuravy It seems that all the items in your list above are carried out. Can you checkmark them all if appropriate?

[ViralBShah]

Should we close this one and focus on #19678 ?

[tkelman]

#19678 will close this when merged.

[ViralBShah]

Ah I missed that #19678 was a PR. Sorry about the noise.