Support LLVM 3.6

[svenstaro]

I know LLVM 3.6 breaks in all kinds of ways because the classical JIT has been removed and now only MCJIT is in there. This is entirely unsupported by OSL right now because the JITMemoryManager is now also gone. I suppose this isn't going to happen to quickly but I think we can use this issue to track that nonetheless.

[mdriftmeyer]

This is sad. LLVM 3.6.2 just got RCd and August LLVM 3.7 OpenMP rolls, but OpenShading is stuck in the past crippling a lot of apps and features, not the least of which is Blender.

[aghiles]

Not sure what you mean by "crippling a lot of apps" here. Certainly not crippling our app. It would be crippling if OSL shaders would run much faster with MCJIT that they do with the current JIT. Would that be the case ?

[lgritz]

It's true that OSL doesn't currently build against LLVM 3.6. I'm working on fixing this ASAP (made a lot of progress on it today, prompted by this ticket).

For those not in the know, here's why it has not been high priority.

• None of the major VFX and animation studios are yet on a C++11 toolchain. They are basically forced into this state by the need to be link compatible with certain 3rd party apps such as Maya and Houdini. Since LLVM/Clang 3.5 requires C++11 to build, that means that none of the major studios -- or software vendors supplying them -- has any incentive or ability to use LLVM newer than 3.4. For now. It is hoped that for 2016, we'll all upgrade to C++11, but for now, nobody in VFX is using it.
• Starting with LLVM 3.6, the "old JIT" has been removed completely, forcing us to use MCJIT. In our experiments, MCJIT is much slower than old JIT, to a point where it's really a big problem for us with lots of complex shaders in a production environment. So even if our toolchain allowed us to use LLVM 3.6 (which it doesn't), at the moment we would very much want to hold back to 3.5 for performance reasons. I'm working on this (quite honestly, very slowly, since it's not a big problem until LLVM 3.6, and we can't even use 3.5 in production yet).

I'm not proud of any of this, but it is what it is. I thought it was worth explaining that it's actually not completely stupid that we are targeting an older version of LLVM.

All that said, there is no excuse for not even building properly against LLVM 3.6 (even if my employers can't use a version that new, and even if it runs slower than would be desired). I want to get this fixed ASAP, and spent some time on it today. I'll try to spend more time on it tomorrow as well and make a PR to add 3.6 support as soon as I have it done.

[aghiles]

Thanks for the explanation. For us (and I think many), shader execution speed is much more important than supporting the latest LLVM version, so as long as 3.5 is supported we are fine.

Now I can see this as a long term problem in which users have to choose between old "faster" JIT and much newer LLVM with more optimizations. Do you think that MCJIT is fundamentally slower for the way OSL works or is jut that it needs more work and could potentially end up as fast as old JIT ?

[lgritz]

I'm not sure, Aghiles. There are so many moving parts! MCJIT improves over time. We learn workarounds (in particular, Animal Logic had done some patches to LLVM that, if we can get them incorporated, may help a lot). Also in 3.7 there is yet ANOTHER jit -- "orc jit" -- which I have not yet investigated to see how it compares with MCJIT. Always lots to try.

I'm not "worried" for the long term. I just need to do the work to make something that's just as fast, on the newer LLVMs. But since the C++11 issue has kept the newer LLVM releases out of reach for us anyway, I just haven't done the leg work on this front yet.

[aghiles]

OK Larry. We are starting to use OSL. So expect some help and reports from our side.

[Olivier-P]

None of the major VFX and animation studios are yet on a C++11 toolchain. They are basically forced into this state by the need to be link compatible with certain 3rd party apps such as Maya and Houdini.

Has anyone actually tried this? I don't know about Houdini but the Maya API is fairly plain C++ which doesn't care much about compiler version. While they claim you need gcc 4.1.2 on linux (4.4.7 with 2016) or the world will implode, we've built our plugins with various compilers over the years (currently 4.4.7) and it works. The biggest problem was back when Maya shipped its own libstdc++.so (your binary had to run with that one) but thankfully they stopped that with maya 2011. The situation appears similar on windows, although we've used fewer compilers there.

If everyone is waiting on Autodesk to move forward, we may be waiting a long long time. They don't have a history of quick changes and given the amount of code they must build, I can understand why they would be conservative with their toolchain.

[lgritz]

Are you following http://www.vfxplatform.com/ ? In theory, Autodesk is on board. In practice, we'll see which versions ship with what.

[ppearson]

We couldn't move to 4.6 for Maya or Nuke, as we discovered compatibility issues, especially with std::string.
The way different compilers seemed to inline things seemed to cause quite a few issues with symbol visibility differences for header-only stuff.

[Alexpux]

Does anyone have dirty patch to allow building with clang 3.6+?

[svenstaro]

@Alexpux there is no dirty patching this time. This just wouldn't work given the large amount of changes.

[lgritz]

I'm actively working on the LLVM 3.6 issue.

[Alexpux]

@lgritz ok, thanks!

[aditya4d1]

@lgritz So, we can only build OSL against LLVM 3.4?

[lgritz]

You can build OSL against any LLVM <= 3.5.

LLVM 3.2 and 3.3 certainly used to work. I have no reason to think they don't still work, but I haven't tested them for a long time.

LLVM 3.4 is what I use daily and in production.

LLVM 3.5 is known to work, and I occasionally test with it, but I don't use it in production because it requires a C++11 toolchain and we aren't (yet) up to that at work.

LLVM >= 3.6 is known not to work because they've changed the JIT APIs. It is on my plate to fix this soon.

[aditya4d1]

Thank you! Can you please comment on the make error I have reported? I am using llvm 3.4

[abergmeier]

@lgritz Is it realistic, that this (the sheer compilation) will get fixed soon?
With my home laptop I cannot do any Blender development anymore due to this.
We probably will have to stop updating our Ubuntu's at work due to this.

[brechtvl]

@abergmeier, the latest Blender source code has script to build dependencies with options to build LLVM and OSL from source, which works on the latest Ubuntu. This LLVM version issue shouldn't stop anyone from doing Blender development, it's mainly an issue for package maintainers.

[reanimastudios]

What does that script to build dependencies have to do with building OpenShadingLanguage with a recent version of LLVM? Sorry, but OSL continues to be broken for anything newer than LLVM 3.5, but only officially supported up to LLVM 3.4.

If you know anything about Debian you know they maintain all their LLVM/Clang lines from LLVM 3.5+. Nothing older.

This wouldn't be a problem if OSL would get off its butt and update the code that has been rotting for over 18 months. LLVM is on a 6-9 month release cycle for each update. LLVM 3.7.1 was just released. The JIT is never going to be without MCJIT. Either crap or get off the pot already.

LLVM 3.8 is nearing release.

[brechtvl]

@mdriftmeyer, all I'm saying is, if you're a Blender user or developer and encounter this problem, then the solution is to follow the official Blender build instructions. The script will build LLVM 3.4 from source and use it for OSL, which works fine regardless of the LLVM package version in your Linux distribution.

This is an open source project, people are busy, you can either contribute or be patient.

[lgritz]

Supporting newer releases of LLVM is certainly on my radar, but it has never been a critical request (or really, a concern at all) from anybody with actual "skin in the game" -- those funding the project by paying my salary or that of other contributors, those using it for large-scale VFX and animation production, and those embedding it in widely-used products (including the main authors of Cycles).

We didn't make LLVM retire the old JIT without a compatibility mode for MCJIT, nor make them abandon compatibility with C++03 (which has made LLVM > 3.4 irrelevant to the VFX industry, which prior to 2016 has been unable to migrate to C++11).

Nonetheless, it is on the list to do. I can't guarantee that we'll support 3.6 specifically; we may jump straight to 3.7+ with ORC JIT. My aim is to have this in time for a stable 1.7 branch at the end of the month. But I can't and won't make any guarantees.

In the mean time, everybody is free to build and use LLVM 3.4 (even if it's not the default LLVM for your system -- its source is downloadable), and everybody is free to figure out MCJIT or ORC themselves and submit a PR.

[sambler]

Just want to wake this up and suggest aiming to support newer than llvm3.6 now.

FreeBSD 11.0 was released Oct 2016 and has clang3.9 as the base system compiler, with clang4.0 now being tested for merging into current for the next major release. FreeBSD 10.3 has clang3.4 in base and will be supported for another 15 months.

While testing the OSL build on a clang4 based system a patch was added to the new libc++ to support compiling llvm_ops.cpp with 3.4.

Within the FreeBSD ports - OSL is one of three ports that still needs llvm/clang < 3.7 - there are 22 that use llvm 3.7-3.9

[lgritz]

There's a PR (#693) that fully brings us up to LLVM 3.9. I'm evaluating it now to make sure there are no important perf regressions.

[marsupial]

BTW, LLVM 4 seems to provides a nice 10% boost over 3.9.

[lgritz]

In the speed of doing the JIT, or the speed of executing the resulting code?

[marsupial]

Overall speed. Unclear which, but I'd lean towards the JIT as if upgrading to clang 3.9->4.0 generated code which was 10% faster, most people would have heard about that!

[svenstaro]

Seems like this can be closed. :P