Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP working llvm-90 #5389

Merged
merged 9 commits into from Jan 25, 2020
Merged

WIP working llvm-90 #5389

merged 9 commits into from Jan 25, 2020

Conversation

Toasterson
Copy link
Contributor

This is a compiling version of LLVM and clang 9 analogous to pkgsrc with patches from pkgsrc

Building this on my build system takes 12 hours!!! I would thus be glad if somebody could take over polishing this.

The option CMAKE_OPTIONS += -DLLVM_LINK_LLVM_DYLIB=ON does not work for us although it seems to wotk for pkgsrc libLTO does not want to link. It is optional as shared libraries are still generated without it but all shrared libraries are not linked against one another but rather statically linked. Which makes them slightly bigger but it at least compiles.

@Mno-hime I have built without ninja as it made no difference in the end result. Feel free to add it again if you want.

Till Wegmueller and others added 2 commits November 12, 2019 13:07
@Mno-hime
Copy link
Contributor

Obsoleted clang-80 component. Publishing fixes. Makefile modernization.

@Mno-hime
Copy link
Contributor

Firefox should be rebuild (#5394) after this one is merged.

@alarcher
Copy link
Contributor

alarcher commented Nov 12, 2019 via email

@Mno-hime
Copy link
Contributor

@Toasterson rust 1.37 (https://github.com/Toasterson/oi-userland/commits/rust-137) fails to build for me with this LLVM:

     Running `/userland/ws/oi-userland/components/developer/rust/build/amd64/build/x86_64-sun-solaris/stage0-codegen/release/build/rustc_llvm-0d412a82206d98c7/build-script-build`
[rustc_llvm 0.0.0] cargo:rerun-if-env-changed=REAL_LIBRARY_PATH_VAR
[rustc_llvm 0.0.0] cargo:rerun-if-env-changed=REAL_LIBRARY_PATH
[rustc_llvm 0.0.0] cargo:rerun-if-changed=/usr/clang/9.0/bin/llvm-config
[rustc_llvm 0.0.0] cargo:rerun-if-env-changed=LLVM_CONFIG
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="aarch64"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="amdgpu"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="arm"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="asmparser"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="bitreader"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="bitwriter"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="hexagon"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="instrumentation"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="interpreter"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="ipo"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="linker"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="lto"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="mcjit"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="mips"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="msp430"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="nvptx"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="powerpc"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="riscv"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="sparc"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="systemz"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="webassembly"
[rustc_llvm 0.0.0] cargo:rustc-cfg=llvm_component="x86"
[rustc_llvm 0.0.0] cargo:rerun-if-changed-env=LLVM_RUSTLLVM
[rustc_llvm 0.0.0] cargo:rerun-if-changed=../rustllvm/rustllvm.h
[rustc_llvm 0.0.0] cargo:rerun-if-changed=../rustllvm/.editorconfig
[rustc_llvm 0.0.0] cargo:rerun-if-changed=../rustllvm/RustWrapper.cpp
[rustc_llvm 0.0.0] cargo:rerun-if-changed=../rustllvm/Linker.cpp
[rustc_llvm 0.0.0] cargo:rerun-if-changed=../rustllvm/README
[rustc_llvm 0.0.0] cargo:rerun-if-changed=../rustllvm/PassWrapper.cpp
[rustc_llvm 0.0.0] cargo:rerun-if-changed=../rustllvm/ArchiveWrapper.cpp
[rustc_llvm 0.0.0] TARGET = Some("x86_64-sun-solaris")
[rustc_llvm 0.0.0] OPT_LEVEL = Some("2")
[rustc_llvm 0.0.0] HOST = Some("x86_64-sun-solaris")
[rustc_llvm 0.0.0] CXX_x86_64-sun-solaris = Some("/usr/gcc/6/bin/g++")
[rustc_llvm 0.0.0] CXXFLAGS_x86_64-sun-solaris = Some("-O2 -ffunction-sections -fdata-sections -fPIC -m64 -m64 -O3 -D_POSIX_PTHREAD_SEMANTICS")
[rustc_llvm 0.0.0] CRATE_CC_NO_DEFAULTS = None
[rustc_llvm 0.0.0] DEBUG = Some("false")
[rustc_llvm 0.0.0] CARGO_CFG_TARGET_FEATURE = Some("fxsr,mmx,sse,sse2")
[rustc_llvm 0.0.0] running: "/usr/gcc/6/bin/g++" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-m64" "-O3" "-D_POSIX_PTHREAD_SEMANTICS" "-I/usr/clang/9.0/include" "-std=c++11" "-fno-exceptions" "-D_DEBUG" "-D__STDC_CONSTANT_MACROS" "-D__STDC_FORMAT_MACROS" "-D__STDC_LIMIT_MACROS" "-DLLVM_COMPONENT_AARCH64" "-DLLVM_COMPONENT_AMDGPU" "-DLLVM_COMPONENT_ARM" "-DLLVM_COMPONENT_ASMPARSER" "-DLLVM_COMPONENT_BITREADER" "-DLLVM_COMPONENT_BITWRITER" "-DLLVM_COMPONENT_HEXAGON" "-DLLVM_COMPONENT_INSTRUMENTATION" "-DLLVM_COMPONENT_INTERPRETER" "-DLLVM_COMPONENT_IPO" "-DLLVM_COMPONENT_LINKER" "-DLLVM_COMPONENT_LTO" "-DLLVM_COMPONENT_MCJIT" "-DLLVM_COMPONENT_MIPS" "-DLLVM_COMPONENT_MSP430" "-DLLVM_COMPONENT_NVPTX" "-DLLVM_COMPONENT_POWERPC" "-DLLVM_COMPONENT_RISCV" "-DLLVM_COMPONENT_SPARC" "-DLLVM_COMPONENT_SYSTEMZ" "-DLLVM_COMPONENT_WEBASSEMBLY" "-DLLVM_COMPONENT_X86" "-o" "/userland/ws/oi-userland/components/developer/rust/build/amd64/build/x86_64-sun-solaris/stage0-codegen/x86_64-sun-solaris/release/build/rustc_llvm-2ac19e32c1d0d622/out/../rustllvm/PassWrapper.o" "-c" "../rustllvm/PassWrapper.cpp"
[rustc_llvm 0.0.0] cargo:warning=../rustllvm/PassWrapper.cpp: In function 'LLVMRustThinLTOData* LLVMRustCreateThinLTOData(LLVMRustThinLTOModule*, int, const char**, int)':
[rustc_llvm 0.0.0] cargo:warning=../rustllvm/PassWrapper.cpp:917:77: error: too few arguments to function 'void llvm::thinLTOResolvePrevailingInIndex(llvm::ModuleSummaryIndex&, llvm::function_ref<bool(long unsigned int, const llvm::GlobalValueSummary*)>, llvm::function_ref<void(llvm::StringRef, long unsigned int, llvm::GlobalValue::LinkageTypes)>, const llvm::DenseSet<long unsigned int>&)'
[rustc_llvm 0.0.0] cargo:warning=   thinLTOResolvePrevailingInIndex(Ret->Index, isPrevailing, recordNewLinkage);
[rustc_llvm 0.0.0] cargo:warning=                                                                             ^
[rustc_llvm 0.0.0] cargo:warning=In file included from ../rustllvm/PassWrapper.cpp:22:0:
[rustc_llvm 0.0.0] cargo:warning=/usr/clang/9.0/include/llvm/LTO/LTO.h:49:6: note: declared here
[rustc_llvm 0.0.0] cargo:warning= void thinLTOResolvePrevailingInIndex(
[rustc_llvm 0.0.0] cargo:warning=      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[rustc_llvm 0.0.0] thread 'main' panicked at '
[rustc_llvm 0.0.0] 
[rustc_llvm 0.0.0] Internal error occurred: Command "/usr/gcc/6/bin/g++" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-m64" "-O3" "-D_POSIX_PTHREAD_SEMANTICS" "-I/usr/clang/9.0/include" "-std=c++11" "-fno-exceptions" "-D_DEBUG" "-D__STDC_CONSTANT_MACROS" "-D__STDC_FORMAT_MACROS" "-D__STDC_LIMIT_MACROS" "-DLLVM_COMPONENT_AARCH64" "-DLLVM_COMPONENT_AMDGPU" "-DLLVM_COMPONENT_ARM" "-DLLVM_COMPONENT_ASMPARSER" "-DLLVM_COMPONENT_BITREADER" "-DLLVM_COMPONENT_BITWRITER" "-DLLVM_COMPONENT_HEXAGON" "-DLLVM_COMPONENT_INSTRUMENTATION" "-DLLVM_COMPONENT_INTERPRETER" "-DLLVM_COMPONENT_IPO" "-DLLVM_COMPONENT_LINKER" "-DLLVM_COMPONENT_LTO" "-DLLVM_COMPONENT_MCJIT" "-DLLVM_COMPONENT_MIPS" "-DLLVM_COMPONENT_MSP430" "-DLLVM_COMPONENT_NVPTX" "-DLLVM_COMPONENT_POWERPC" "-DLLVM_COMPONENT_RISCV" "-DLLVM_COMPONENT_SPARC" "-DLLVM_COMPONENT_SYSTEMZ" "-DLLVM_COMPONENT_WEBASSEMBLY" "-DLLVM_COMPONENT_X86" "-o" "/userland/ws/oi-userland/components/developer/rust/build/amd64/build/x86_64-sun-solaris/stage0-codegen/x86_64-sun-solaris/release/build/rustc_llvm-2ac19e32c1d0d622/out/../rustllvm/PassWrapper.o" "-c" "../rustllvm/PassWrapper.cpp" with args "g++" did not execute successfully (status code exit code: 1).
[rustc_llvm 0.0.0] exit code: 1
[rustc_llvm 0.0.0] 
[rustc_llvm 0.0.0] ', /userland/ws/oi-userland/components/developer/rust/build/amd64/vendor/cc/src/lib.rs:2398:5
[rustc_llvm 0.0.0] stack backtrace:
[rustc_llvm 0.0.0]    0: std::sys::unix::backtrace::tracing::imp::unwind_backtrace
[rustc_llvm 0.0.0]    1: std::sys_common::backtrace::print
[rustc_llvm 0.0.0]    2: std::panicking::default_hook::{{closure}}
[rustc_llvm 0.0.0]    3: std::panicking::default_hook
[rustc_llvm 0.0.0]    4: std::panicking::rust_panic_with_hook
[rustc_llvm 0.0.0]    5: std::panicking::continue_panic_fmt
[rustc_llvm 0.0.0]    6: std::panicking::begin_panic_fmt
[rustc_llvm 0.0.0]    7: cc::fail
[rustc_llvm 0.0.0]    8: cc::Build::compile
[rustc_llvm 0.0.0]    9: build_script_build::main
[rustc_llvm 0.0.0]   10: std::rt::lang_start::{{closure}}
[rustc_llvm 0.0.0]   11: std::panicking::try::do_call
[rustc_llvm 0.0.0]   12: __rust_maybe_catch_panic
[rustc_llvm 0.0.0]   13: std::rt::lang_start_internal
[rustc_llvm 0.0.0]   14: main
[rustc_llvm 0.0.0]   15: _start_crt
[rustc_llvm 0.0.0]   16: _start

Any ideas?

@Mno-hime
Copy link
Contributor

Any chance to leave clang-80 for now?

Sure. Any specific reason? In the past we shipped only on LLVM/Clang conponent.

@Toasterson
Copy link
Contributor Author

Toasterson commented Nov 12, 2019

@Mno-hime edit: sorry just saw that it is trying with the external llvm. This has not been updated before trying clang. So you probably need patches from pkgsrc and other stuff.

@alarcher
Copy link
Contributor

Sure. Any specific reason? In the past we shipped only on LLVM/Clang component.

I use clang-80 for scientific computing projects (also scanbuild etc..) so if you remove the shared libs this is an issue for me. I am a real consumer of the compiler.

@Mno-hime
Copy link
Contributor

Sure. Any specific reason? In the past we shipped only on LLVM/Clang component.

I use clang-80 for scientific computing projects (also scanbuild etc..) so if you remove the shared libs this is an issue for me. I am a real consumer of the compiler.

I see, sorry about that, clang-80 is back.

@Mno-hime
Copy link
Contributor

@Mno-hime edit: sorry just saw that it is trying with the external llvm. This has not been updated before trying clang. So you probably need patches from pkgsrc and other stuff.

I see thanks.

@Mno-hime
Copy link
Contributor

This builds for me, perhaps we can merge and start working on newer Rust?

@Toasterson
Copy link
Contributor Author

I would say so.

@Toasterson
Copy link
Contributor Author

Is something still missing other than an LGTM from me? (it's my pr so i can't review via Github)

@pyhalov
Copy link
Contributor

pyhalov commented Dec 11, 2019

Please, look through existing patches.

  1. Check that _REENTRANT is set when -pthreads is used (not -pthread), GCC compat (look at https://github.com/OpenIndiana/oi-userland/blob/oi/hipster/components/developer/clang-80/patches/05-pthreads.patch).
  2. Check that rpath are properly passed to the linker (look at https://github.com/OpenIndiana/oi-userland/blob/oi/hipster/components/developer/clang-80/patches/04-rpath.patch)
  3. add https://github.com/OpenIndiana/oi-userland/blob/oi/hipster/components/developer/clang-80/patches/02-gas.patch, we don't want llmv to call Sun as

Copy link
Contributor

@pyhalov pyhalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, add missing patches from current clang.

@cgrzemba
Copy link
Contributor

rust is not building for me because a of missing shared llvm libs, e.g. libLLVM-[789].so
pkgsrc delivers llvm sepearte from clang.

@Mno-hime
Copy link
Contributor

@pyhalov cd519a7 brings those patches back.

@Mno-hime
Copy link
Contributor

Mno-hime commented Jan 6, 2020

rust is not building for me because a of missing shared llvm libs, e.g. libLLVM-[789].so
pkgsrc delivers llvm sepearte from clang.

@cgrzemba This PR provides these libraries:

$ pkg contents clang-90 | grep libLLVM.*.so
usr/clang/9.0/lib/libLLVM-9.0.0.so
usr/clang/9.0/lib/libLLVM-9.so
usr/clang/9.0/lib/libLLVM.so

But still I can't build Rust (of any version) with it:

running: "/usr/gcc/6/bin/g++" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-O3" "-D_POSIX_PTHREAD_SEMANTICS" "-m64" "-m64" "-I/usr/clang/9.0/include" "-std=c++11" "-fno-exceptions" "-D_DEBUG" "-D__STDC_CONSTANT_MACROS" "-D__STDC_FORMAT_MACROS" "-D__STDC_LIMIT_MACROS" "-DLLVM_COMPONENT_AARCH64" "-DLLVM_COMPONENT_AMDGPU" "-DLLVM_COMPONENT_ARM" "-DLLVM_COMPONENT_ASMPARSER" "-DLLVM_COMPONENT_BITREADER" "-DLLVM_COMPONENT_BITWRITER" "-DLLVM_COMPONENT_HEXAGON" "-DLLVM_COMPONENT_INSTRUMENTATION" "-DLLVM_COMPONENT_INTERPRETER" "-DLLVM_COMPONENT_IPO" "-DLLVM_COMPONENT_LINKER" "-DLLVM_COMPONENT_LTO" "-DLLVM_COMPONENT_MCJIT" "-DLLVM_COMPONENT_MIPS" "-DLLVM_COMPONENT_MSP430" "-DLLVM_COMPONENT_NVPTX" "-DLLVM_COMPONENT_POWERPC" "-DLLVM_COMPONENT_RISCV" "-DLLVM_COMPONENT_SPARC" "-DLLVM_COMPONENT_SYSTEMZ" "-DLLVM_COMPONENT_WEBASSEMBLY" "-DLLVM_COMPONENT_X86" "-o" "/userland/ws/oi-userland/components/developer/rust/build/amd64/build/x86_64-sun-solaris/stage0-codegen/x86_64-sun-solaris/release/build/rustc_llvm-2b35a807e827cf30/out/../rustllvm/PassWrapper.o" "-c" "../rustllvm/PassWrapper.cpp"
cargo:warning=../rustllvm/PassWrapper.cpp: In function 'LLVMRustThinLTOData* LLVMRustCreateThinLTOData(LLVMRustThinLTOModule*, int, const char**, int)':
cargo:warning=../rustllvm/PassWrapper.cpp:926:77: error: too few arguments to function 'void llvm::thinLTOResolvePrevailingInIndex(llvm::ModuleSummaryIndex&, llvm::function_ref<bool(long unsigned int, const llvm::GlobalValueSummary*)>, llvm::function_ref<void(llvm::StringRef, long unsigned int, llvm::GlobalValue::LinkageTypes)>, const llvm::DenseSet<long unsigned int>&)'
cargo:warning=   thinLTOResolvePrevailingInIndex(Ret->Index, isPrevailing, recordNewLinkage);
cargo:warning=                                                                             ^
cargo:warning=In file included from ../rustllvm/PassWrapper.cpp:38:0:
cargo:warning=/usr/clang/9.0/include/llvm/LTO/LTO.h:49:6: note: declared here
cargo:warning= void thinLTOResolvePrevailingInIndex(
cargo:warning=      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

@Toasterson
Copy link
Contributor Author

@Mno-hime which rust version are you trying to build? according to the Error Rust stumbles over an API change in libLTO (LLVM) our currently used version 1.32 will not build we will need to build at least 1.39 or later.

@Mno-hime
Copy link
Contributor

Mno-hime commented Jan 8, 2020

@Toasterson I think that was 1.32.0 (the one we already ship), the point of the exercise is to see if the updated LLVM 9.0 is able to build the Rust we already have and then move to newer Rust versions.

I identified rust-lang/rust@04304fc as a way to pass this error. But that leads to another errors: https://paste.ec/paste/OOXJGm3p#tDKt0dlKIhadlj+E7C846jM7FuzJl1Jm3YZQ1v1E3wz.

Maybe we should fix LLVM 8.0 instead and build Rust with it.

@Toasterson
Copy link
Contributor Author

Knowing Rust and LLVM the API's both use are not stable at all. So you will need a version of Rust that is compatible with this exact version of LLVM. My guess is 1.32 is not compatible.

LLVM does not help us here.... The Request that set off this rabbit hole was from @jclulow . He and others want to get rust into illumos directly so that components of illumos can be written in rust. And that was for rust current.

I would skip 1.32 it's too old to be worth it.

CmdArgs.push_back(II.getFilename());

- const char *Exec = Args.MakeArgString(getToolChain().GetProgramPath("as"));
+ const char *Exec = Args.MakeArgString(getToolChain().GetProgramPath("gas"));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just curious, SPARC uses GNU as now? All IX86 targets supported by clang have long since used the integrated assembler, this statement has no effect as a result

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps, this can be dropped, but this needs some testing.

@Mno-hime
Copy link
Contributor

Mno-hime commented Jan 8, 2020

LLVM does not help us here.... The Request that set off this rabbit hole was from @jclulow . He and others want to get rust into illumos directly so that components of illumos can be written in rust. And that was for rust current.

This and newer Firefox, about which I am currently more interested in.

I would skip 1.32 it's too old to be worth it.

Well, yes, I just hoped for a base line to see if we got the LLVM side right...

I tried many other Rust versions up to 1.40.0 and all of them died somewhere. Some of them, I think telling me my local Rust did not support this or that construct of the Rust I was building... This is so hopeless that I seriously consider reading the rustc development manual :).

@Toasterson
Copy link
Contributor Author

Yes you also need the correct rust to bootstrap the correct rust.

@pyhalov
Copy link
Contributor

pyhalov commented Jan 23, 2020

Seems to be fine, why can't we merge it?

@Mno-hime
Copy link
Contributor

Seems to be fine, why can't we merge it?

I think we can. But technically you request changes to this PR down here :).

@pyhalov
Copy link
Contributor

pyhalov commented Jan 24, 2020

@Toasterson Test results are missing

@Toasterson
Copy link
Contributor Author

@pyhalov Yes I never Managed to compile the latest version due to My machine being to slow. Running through the Suite will take me roughly 48 hours. If @Mno-hime could do It that would be best otherwise I could run it through the weekend if needed.

@pyhalov
Copy link
Contributor

pyhalov commented Jan 25, 2020

I'll merge this when check FF rebuild.

@pyhalov
Copy link
Contributor

pyhalov commented Jan 25, 2020

clang 9.0.1 is available

@pyhalov
Copy link
Contributor

pyhalov commented Jan 25, 2020

Have to update test results.

@pyhalov pyhalov merged commit 5855e58 into OpenIndiana:oi/hipster Jan 25, 2020
@Toasterson
Copy link
Contributor Author

Thanks for the work much appreciated.

@Toasterson Toasterson deleted the clang9 branch January 25, 2020 11:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
6 participants