Request: Update libc++ to LLVM 13

[Urfoex]

LLVM 13.0.0 got released on 4 Oct 2021

Would it be possible to update to it?

I would currently be most interested in
"The implementation of the C++20 concepts library has been completed."

Older tickets:
c++ 20 concept support
concepts not supported

[kripken]

Updated title to focus on libc++, as this is special to that (the emsdk already uses newer LLVM, but not libc++ yet).

[kripken]

If someone is interested to work on this, Emscripten has a bundled version of libc++ in system/lib/libcxx/. I believe to upgrade libc++ the first step would be to open a PR with LLVM 13's libc++ replaced there, and then to work through any issues with tests if there are any.

[sbc100]

This change should probably also include updates to compiler-rt and libc++-abi as well. There are some scripts that should make the process easier:
https://github.com/emscripten-core/emscripten/blob/main/system/lib/update_libcxx.py
https://github.com/emscripten-core/emscripten/blob/main/system/lib/update_compiler_rt.py
https://github.com/emscripten-core/emscripten/blob/main/system/lib/update_libcxxabi.py

[Urfoex]

This change should probably also include updates to compiler-rt and libc++-abi as well. There are some scripts that should make the process easier: https://github.com/emscripten-core/emscripten/blob/main/system/lib/update_libcxx.py https://github.com/emscripten-core/emscripten/blob/main/system/lib/update_compiler_rt.py https://github.com/emscripten-core/emscripten/blob/main/system/lib/update_libcxxabi.py

Looking at those python scripts, they just copy files. I suspect, that there need some patches to be applied, or reapplied afterwards, as pointed out in e.g. https://github.com/emscripten-core/emscripten/blob/main/system/lib/libcxx/readme.txt

[kripken]

Yes, it may make sense to look at the diff before, then apply it afterwards.

[sbc100]

My idea (which I admit is still being thought through, and not documented) is that all the patches for llvm libraries will live in our llvm fork at: https://github.com/emscripten-core/llvm-project

I do mention this in the readme for libcxx, libcxxabi, compiler-rt, but I don't talk about how we maintain them there:

emscripten/system/lib/compiler-rt/readme.txt

Lines 1 to 9 in 0d24418

	llvm's compiler-rt
	------------------
	These files are from the llvm-project based on release 12.0.0.
	We maintain a local fork of llvm-project that contains any emscripten
	specific patches:
	https://github.com/emscripten-core/llvm-project

The idea is that for each major llvm release we maintain a branch in our llvm fork, and we periodically update it by copying the emscipten-files back to the fork. For example, for llvm 12.0.0 I we a branch called emscripten-libs-12.0.0. Comparing this branch with llvmorg-12.0.0 gives is the complete set of emscripten patches for llvm 12.0.0:

emscripten-core/llvm-project@llvmorg-12.0.0...emscripten-core:emscripten-libs-12.0.0

The idea is that when one wants to upgrade to llvm-13 one could rebase all of the llvm-12 changes onto the llvm-13 branch. At this point any conflicts could be resolved and the resulting branch in the llvm repo could then be copied back into the emscripten repo.

I'll see if I can take care of documenting/automating/improving this process as part of the llvm-13 upgrade,

[Urfoex]

Tried building LLVM.
Run into some RAM problems when linking.
Changing the linker from ld to ldd helped a lot. (Gold might also work.)
Release instead of Debug might also have helped.

CMAKE_BUILD_TYPE: Release
CMAKE_LINKER: /usr/bin/ld.lld
GOLD_EXECUTABLE: /usr/bin/ld.lld

I created a pull request against main to update it to latest upstream. (If that was ok. Sounded sensible before branching off its tag: llvmorg-13.0.0 )

[sbc100]

Hopefully updating libc++/libc++abi/compiler-rt in emscripten will not actually require the building of llvm. Is there some reason you need to build it that I'm missing?

[Urfoex]

I guess I just had all projects enabled.
This one should do better in my case:
cmake -S llvm -B build -G Ninja -DLLVM_ENABLE_PROJECTS='clang;compiler-rt;libcxx;libcxxabi' -DCMAKE_LINKER=/usr/bin/ld.lld -DGOLD_EXECUTABLE=/usr/bin/ld.lld -DCMAKE_BUILD_TYPE=Debug
or
cmake -S llvm -B build -G Ninja -DLLVM_ENABLE_PROJECTS='clang;compiler-rt;libcxx;libcxxabi' -DCMAKE_LINKER=/usr/bin/ld.lld -DGOLD_EXECUTABLE=/usr/bin/ld.lld -DCMAKE_BUILD_TYPE=Release

cmake --build build
cmake --build build -- check-all

[sbc100]

There should be no need to configure or build llvm to update emscripten's library code. IIUC, all compiling and testing of these library copies is done in the emscripten tree. The llvm tree is just the place where the upstream sources live, and where we hope to one day upstream all out changes (#15097)

[Urfoex]

You don't want to at least test compile the code to see if it might has problems?

Ah!
I see.
Inside the emscripten tree.
That would make sense.
As I was now missing out on wasi/api.h. And installing wasi-libc didn't directly help...

[sbc100]

I'm afraid there might be a lot of internal implementation of knowledge required to do this update. The process is (sadly) not well documented. Unless you are keen to learn a lot of how all this fits together it might not be good task to take on as new contributor.

The wasi/api.h file should remain intact during this update, it comes from wasi-sdk originally. The file lives in system/include/wasi/api.h which should not be effected by this change.

[Urfoex]

I'm afraid there might be a lot of internal implementation of knowledge required to do this update. The process is (sadly) not well documented. Unless you are keen to learn a lot of how all this fits together it might not be good task to take on as new contributor.

I'm afraid you are very much right.
I dug through emsdk and emscripten, trying to get the tests running, as I thought, maybe after copying the new lib-versions over it would be good to see, that it still works. But I also didn't get them to work. Step by step hitting one roadblock after another. Frustrating. So, I'll better stop here and wait for some experienced people to solve this. Sorry.