Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error while compile firefox - crash backtrace for analysis #87894

Open
Cerberios opened this issue Apr 6, 2024 · 30 comments
Open

Error while compile firefox - crash backtrace for analysis #87894

Cerberios opened this issue Apr 6, 2024 · 30 comments
Labels
crash Prefer [crash-on-valid] or [crash-on-invalid] llvm:SLPVectorizer

Comments

@Cerberios
Copy link

Cerberios commented Apr 6, 2024

Hello there! I am not a programmer, but got this error - please, investigate, may be it will help to make product better.
I used sys-devel/llvm 17.0.6 + polly
System uname: Linux-6.8.4-gentoo-simple-gcc-x86_64-13th_Gen_Intel-R-_Core-TM-_i5-1335U-with-glibc2.38
python 3.11.9-final-0, default/linux/amd64/23.0/split-usr/hardened, gcc-13, glibc-2.38-r11, 6.8.4-gentoo-simple-gcc x86_64

Please, let me know what kind of another logs (settings) I should provide.

14:07.13 PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace. 14:07.20 #0 0x00007bb5bcd68406 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xd68406) 14:07.20 #1 0x00007bb5bcd652af llvm::sys::RunSignalHandlers() (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xd652af) 14:07.20 #2 0x00007bb5bcd653fd (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xd653fd) 14:07.20 #3 0x00007bb5c744d2e0 (/lib64/libc.so.6+0x3e2e0) 14:07.20 #4 0x00007bb5be78969f llvm::slpvectorizer::BoUpSLP::vectorizeTree(llvm::slpvectorizer::BoUpSLP::TreeEntry*) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x278969f) 14:07.20 #5 0x00007bb5be78cfa4 llvm::slpvectorizer::BoUpSLP::vectorizeOperand(llvm::slpvectorizer::BoUpSLP::TreeEntry*, unsigned int) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x278cfa4) 14:07.20 #6 0x00007bb5be78a611 llvm::slpvectorizer::BoUpSLP::vectorizeTree(llvm::slpvectorizer::BoUpSLP::TreeEntry*) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x278a611) 14:07.21 #7 0x00007bb5be78d994 llvm::slpvectorizer::BoUpSLP::vectorizeTree(llvm::MapVector<llvm::Value*, llvm::SmallVector<llvm::Instruction*, 2u>, llvm::DenseMap<llvm::Value*, unsigned int, llvm::DenseMapInfo<llvm::Value*, void>, llvm::detail::DenseMapPair<llvm::Value*, unsigned int>>, llvm::SmallVector<std::pair<llvm::Value*, llvm::SmallVector<llvm::Instruction*, 2u>>, 0u>> const&, llvm::SmallVectorImpl<std::pair<llvm::Value*, llvm::Value*>>&, llvm::Instruction*) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x278d994) 14:07.21 #8 0x00007bb5be790d25 llvm::slpvectorizer::BoUpSLP::vectorizeTree() (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x2790d25) 14:07.21 #9 0x00007bb5be79bef8 llvm::SLPVectorizerPass::tryToVectorizeList(llvm::ArrayRef<llvm::Value*>, llvm::slpvectorizer::BoUpSLP&, bool) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x279bef8) 14:07.21 #10 0x00007bb5be79e931 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x279e931) 14:07.21 #11 0x00007bb5be7a973d llvm::SLPVectorizerPass::vectorizeChainsInBlock(llvm::BasicBlock*, llvm::slpvectorizer::BoUpSLP&) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x27a973d) 14:07.21 #12 0x00007bb5be7ab3d0 llvm::SLPVectorizerPass::runImpl(llvm::Function&, llvm::ScalarEvolution*, llvm::TargetTransformInfo*, llvm::TargetLibraryInfo*, llvm::AAResults*, llvm::LoopInfo*, llvm::DominatorTree*, llvm::AssumptionCache*, llvm::DemandedBits*, llvm::OptimizationRemarkEmitter*) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x27ab3d0) 14:07.22 #13 0x00007bb5be7abbcc llvm::SLPVectorizerPass::run(llvm::Function&, llvm::AnalysisManager<llvm::Function>&) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x27abbcc) 14:07.22 #14 0x00007bb5c07ce4f6 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x47ce4f6) 14:07.22 #15 0x00007bb5bf7ab098 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x37ab098) 14:07.22 #16 0x00007bb5bcf6b92e llvm::ModuleToFunctionPassAdaptor::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xf6b92e) 14:07.22 #17 0x00007bb5bf78ebb6 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x378ebb6) 14:07.22 #18 0x00007bb5bcf69968 llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xf69968) 14:07.22 #19 0x00007bb5bec398e1 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x2c398e1) 14:07.22 #20 0x00007bb5bec3cb4e (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x2c3cb4e) 14:07.23 #21 0x00007bb5bec3d89a llvm::lto::thinBackend(llvm::lto::Config const&, unsigned int, std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>, llvm::Module&, llvm::ModuleSummaryIndex const&, llvm::StringMap<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::MallocAllocator> const&, llvm::DenseMap<unsigned long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long, void>, llvm::detail::DenseMapPair<unsigned long, llvm::GlobalValueSummary*>> const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, llvm::SmallVector<std::pair<llvm::StringRef, llvm::BitcodeModule>, 0u>>*, std::vector<unsigned char, std::allocator<unsigned char>> const&) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x2c3d89a) 14:07.23 #22 0x00007bb5bec200af (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x2c200af) 14:07.23 #23 0x00007bb5bec2fd9d (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x2c2fd9d) 14:07.23 #24 0x00007bb5bcc1cd16 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xc1cd16) 14:07.23 #25 0x00007bb5bcce9e96 llvm::ThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xce9e96) 14:07.23 #26 0x00007bb5bccea8d7 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xcea8d7) 14:07.23 #27 0x00007bb5c749b9c4 (/lib64/libc.so.6+0x8c9c4) 14:07.23 #28 0x00007bb5c750f1cc (/lib64/libc.so.6+0x1001cc) 14:07.30 Failed to hack libxul.so: basic_ios::clear: iostream error 14:07.30 x86_64-pc-linux-gnu-clang++-17: error: linker command failed with exit code 1 (use -v to see invocation) 14:07.31 gmake[4]: *** [/var/tmp/notmpfs/portage/www-client/firefox-124.0.2/work/firefox-124.0.2/config/rules.mk:541: libxul.so] Error 1 14:07.31 gmake[4]: Leaving directory '/var/tmp/notmpfs/portage/www-client/firefox-124.0.2/work/firefox_build/toolkit/library/build' 14:07.31 gmake[3]: *** [/var/tmp/notmpfs/portage/www-client/firefox-124.0.2/work/firefox-124.0.2/config/recurse.mk:72: toolkit/library/build/target] Error 2 14:07.31 gmake[3]: Leaving directory '/var/tmp/notmpfs/portage/www-client/firefox-124.0.2/work/firefox_build' 14:07.31 gmake[2]: *** [/var/tmp/notmpfs/portage/www-client/firefox-124.0.2/work/firefox-124.0.2/config/recurse.mk:34: compile] Error 2 14:07.31 gmake[2]: Leaving directory '/var/tmp/notmpfs/portage/www-client/firefox-124.0.2/work/firefox_build' 14:07.31 gmake[1]: *** [/var/tmp/notmpfs/portage/www-client/firefox-124.0.2/work/firefox-124.0.2/config/rules.mk:361: default] Error 2 14:07.31 gmake[1]: Leaving directory '/var/tmp/notmpfs/portage/www-client/firefox-124.0.2/work/firefox_build' 14:07.31 gmake: *** [client.mk:60: build] Error 2

Thanks!

@EugeneZelenko EugeneZelenko added llvm:SLPVectorizer crash Prefer [crash-on-valid] or [crash-on-invalid] and removed new issue labels Apr 6, 2024
@EugeneZelenko
Copy link
Contributor

Could you please try 18 or main branch?

@Cerberios
Copy link
Author

Yes, sure just need compile bunch of packages. Will report asap

@alexey-bataev
Copy link
Member

Yes, sure just need compile bunch of packages. Will report asap

Hi, thanks for the report. Compiler saves 2 files - sh script and the preprocessed source code file, which cause the crass (the names are reported in the end of the crash report). Could you, please, copy them and attach them to this bug report? It will help to investigate the crash and will save lots of time.

@dememax
Copy link

dememax commented Apr 19, 2024

@alexey-bataev What are those files?
At least, what are their extensions exactly?
I'm attaching more longer file with my crash.
firefox-llvm-17-crash.txt

@dememax
Copy link

dememax commented Apr 19, 2024

By searching in Gentoo bugs, I've found https://bugs.gentoo.org/925101
And the link to #84062 from there.

@alexey-bataev
Copy link
Member

alexey-bataev commented Apr 19, 2024

@alexey-bataev What are those files? At least, what are their extensions exactly? I'm attaching more longer file with my crash. firefox-llvm-17-crash.txt

Can you try to invoke the compiler manually and add -Wl,-plugin-opt=-print-on-crash -mllvm -print-module-scope -Wl,-plugin-opt=-print-on-crash-path=repro.ll?
I assume it should look like this:

/usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -march=native -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o  -B /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack -Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -Wl,-plugin-opt=-import-hot-multiplier=30 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,--undefined-version -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -fstack-protector-strong -fstack-clash-protection -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -Wl,-rpath-link,/usr/lib  ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a  -Wl,--version-script,libipcclientcerts.so.symbols�-Wl,-plugin-opt=-print-on-crash -mllvm -print-module-scope -Wl,-plugin-opt=-print-on-crash-path=repro.ll

It should produce repro.ll file, that causes the crash

@dememax
Copy link

dememax commented Apr 19, 2024

Yes, it will take a while.
In fact, I've solved my problem by switching off -pgo, but I can rebuild to have locally the broken build, it is reproducible.
Can you contact me in person to instruct in the case this command will be incomplete or fail?

@alexey-bataev
Copy link
Member

Yes, it will take a while. In fact, I've solved my problem by switching off -pgo, but I can rebuild to have locally the broken build, it is reproducible. Can you contact me in person to instruct in the case this command will be incomplete or fail?

Sure, let's see if this one works

@dememax
Copy link

dememax commented Apr 20, 2024

Here:

maxmsi /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/instrumented/security/manager/ssl/ipcclientcerts/dynamic-library # /usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -march=native -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o  -B /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack -Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -Wl,-plugin-opt=-import-hot-multiplier=30 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,--undefined-version -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -fstack-protector-strong -fstack-clash-protection -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -Wl,-rpath-link,/usr/lib  ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a  -Wl,--version-script,libipcclientcerts.so.symbols -Wl,-plugin-opt=-print-on-crash -mllvm -print-module-scope -Wl,-plugin-opt=-print-on-crash-path=repro.ll
x86_64-pc-linux-gnu-clang++-17: warning: argument unused during compilation: '-mllvm -print-module-scope' [-Wunused-command-line-argument]
ld.lld: error: undefined symbol: __llvm_profile_set_timestamp
>>> referenced by stub.cpp
>>>               stub.o:(C_GetFunctionList)
x86_64-pc-linux-gnu-clang++-17: error: linker command failed with exit code 1 (use -v to see invocation)
maxmsi /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/instrumented/security/manager/ssl/ipcclientcerts/dynamic-library # ls -lt
total 20
-rw-r--r-- 1 portage portage 2392 Apr 19 19:37 stub.o
-rw-r--r-- 1 portage portage   66 Apr 19 19:35 libipcclientcerts.so.symbols
-rw-r--r-- 1 portage portage 6464 Apr 19 18:59 backend.mk
-rw-r--r-- 1 portage portage  631 Apr 19 18:59 Makefile

@dememax
Copy link

dememax commented Apr 20, 2024

With -v:

maxmsi /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/instrumented/security/manager/ssl/ipcclientcerts/dynamic-library # /usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -march=native -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o  -B /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack -Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -Wl,-plugin-opt=-import-hot-multiplier=30 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,--undefined-version -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -fstack-protector-strong -fstack-clash-protection -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -Wl,-rpath-link,/usr/lib  ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a  -Wl,--version-script,libipcclientcerts.so.symbols -Wl,-plugin-opt=-print-on-crash -mllvm -print-module-scope -Wl,-plugin-opt=-print-on-crash-path=repro.ll -v
clang version 17.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm/17/bin
Configuration file: /etc/clang/x86_64-pc-linux-gnu-clang++.cfg
System configuration file directory: /etc/clang
Selected GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/14
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
x86_64-pc-linux-gnu-clang++-17: warning: argument unused during compilation: '-mllvm -print-module-scope' [-Wunused-command-line-argument]
 "/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack/ld.lld" --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -shared -o libipcclientcerts.so /usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../lib64/crti.o /usr/lib/gcc/x86_64-pc-linux-gnu/14/crtbeginS.o -L/usr/lib/gcc/x86_64-pc-linux-gnu/14 -L/usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../x86_64-pc-linux-gnu/lib -L/lib -L/usr/lib -plugin-opt=mcpu=skylake -plugin-opt=O3 -plugin-opt=thinlto -plugin-opt=-function-sections=1 -plugin-opt=-data-sections=1 -z relro -z now -z defs --gc-sections -h libipcclientcerts.so stub.o --real-linker /usr/lib/llvm/17/bin/ld.lld -plugin-opt=-import-instr-limit=10 -plugin-opt=-import-hot-multiplier=30 -lpthread -O1 --as-needed --undefined-version --compress-debug-sections=zlib -rpath=/usr/lib64/firefox --enable-new-dtags -z noexecstack -z text -z relro -z now -z nocopyreloc -rpath-link /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -rpath-link /usr/lib ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a --version-script libipcclientcerts.so.symbols -plugin-opt=-print-on-crash -plugin-opt=-print-on-crash-path=repro.ll -lstdc++ -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-pc-linux-gnu/14/crtendS.o /usr/lib/gcc/x86_64-pc-linux-gnu/14/../../../../lib64/crtn.o
ld.lld: error: undefined symbol: __llvm_profile_set_timestamp
>>> referenced by stub.cpp
>>>               stub.o:(C_GetFunctionList)
x86_64-pc-linux-gnu-clang++-17: error: linker command failed with exit code 1 (use -v to see invocation)

In fact, seeing multilib in this output, I've got two hosts with Gentoo, and on one host it compiles, but on an other - it fails.
The difference is in gentoo profiles:

  • default/linux/amd64/23.0/no-multilib - it compiles and works
  • default/linux/amd64/17.1/desktop - it fails.

So, maybe the key of the problem is this multilib/no- problem.

@dememax
Copy link

dememax commented Apr 20, 2024

@alexey-bataev should we switch to #84062 ? This issue is closed.

@alexey-bataev alexey-bataev reopened this Apr 20, 2024
@alexey-bataev
Copy link
Member

I reopened it, we can continue discussion here. Will check your info on Monday.

@12101111
Copy link

So, maybe the key of the problem is this multilib/no- problem.

My system is musl based and don't support multilib.

Since the workaround disables gpu acceleration when pgo, this bug may be related to the crashed process writing incorrect pgo profile files

https://raw.githubusercontent.com/canonical/firefox-snap/stable/patches/pgo-with-software-webrender.patch

@alexey-bataev
Copy link
Member

alexey-bataev commented Apr 22, 2024

Sorry, provided a bit incorrect flags. Here is the right one:

/usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -march=native -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o  -B /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack -Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -Wl,-plugin-opt=-import-hot-multiplier=30 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,--undefined-version -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -fstack-protector-strong -fstack-clash-protection -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -Wl,-rpath-link,/usr/lib  ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a  -Wl,--version-script,libipcclientcerts.so.symbols -Wl,-plugin-opt=-print-on-crash -Wl,-plugin-opt=-print-module-scope -Wl,-plugin-opt=-print-on-crash-path=repro.ll

Need to use -Wl,-plugin-opt= instead of -mllvm for -print-module-scope just like for all other options

@dememax
Copy link

dememax commented Apr 22, 2024

maxmsi /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/instrumented/security/manager/ssl/ipcclientcerts/dynamic-library # /usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -march=native -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o  -B /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack -Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -Wl,-plugin-opt=-import-hot-multiplier=30 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,--undefined-version -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -fstack-protector-strong -fstack-clash-protection -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -Wl,-rpath-link,/usr/lib  ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a  -Wl,--version-script,libipcclientcerts.so.symbols -Wl,-plugin-opt=-print-on-crash -Wl,-plugin-opt=-print-module-scope -Wl,-plugin-opt=-print-on-crash-path=repro.ll
ld.lld: error: undefined symbol: __llvm_profile_set_timestamp
>>> referenced by stub.cpp
>>>               stub.o:(C_GetFunctionList)
x86_64-pc-linux-gnu-clang++-17: error: linker command failed with exit code 1 (use -v to see invocation)

@alexey-bataev
Copy link
Member

maxmsi /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/instrumented/security/manager/ssl/ipcclientcerts/dynamic-library # /usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -march=native -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o  -B /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack -Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -Wl,-plugin-opt=-import-hot-multiplier=30 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,--undefined-version -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -fstack-protector-strong -fstack-clash-protection -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -Wl,-rpath-link,/usr/lib  ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a  -Wl,--version-script,libipcclientcerts.so.symbols -Wl,-plugin-opt=-print-on-crash -Wl,-plugin-opt=-print-module-scope -Wl,-plugin-opt=-print-on-crash-path=repro.ll
ld.lld: error: undefined symbol: __llvm_profile_set_timestamp
>>> referenced by stub.cpp
>>>               stub.o:(C_GetFunctionList)
x86_64-pc-linux-gnu-clang++-17: error: linker command failed with exit code 1 (use -v to see invocation)

Check if there is a file repro.ll

@dememax
Copy link

dememax commented Apr 22, 2024

@alexey-bataev Sorry, I haven't added, but no repro.ll.

@alexey-bataev
Copy link
Member

So, maybe it crashes on another linking command. Need to rebuild the firefox with the correct flags (-Wl,-plugin-opt=)

@dememax
Copy link

dememax commented Apr 22, 2024

So, maybe it crashes on another linking command. Need to rebuild the firefox with the correct flags (-Wl,-plugin-opt=)

Unfortunately, I don't have enough knowledge to do this. I've never touched firefox build system.

@alexey-bataev
Copy link
Member

So, maybe it crashes on another linking command. Need to rebuild the firefox with the correct flags (-Wl,-plugin-opt=)

Unfortunately, I don't have enough knowledge to do this. I've never touched firefox build system.

No, I don't ask you to do it. Can you reproduce a crash manually?

@dememax
Copy link

dememax commented Apr 22, 2024

What exactly should I run?
ld.lld looks fine for me:
ld.lld--help.txt
ldd-ld.lld.txt

@alexey-bataev
Copy link
Member

I'm just asking if you can reproduce the crash manually. If you run /usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstack-clash-protection -fno-sized-deallocation -fno-aligned-new -march=native -fno-exceptions -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -O3 -fomit-frame-pointer -funwind-tables -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libipcclientcerts.so -o libipcclientcerts.so stub.o -B /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack -Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld -flto=thin -Wl,-plugin-opt=-import-instr-limit=10 -Wl,-plugin-opt=-import-hot-multiplier=30 -lpthread -Wl,-O1 -Wl,--as-needed -Wl,--undefined-version -Wl,--compress-debug-sections=zlib -Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags -fuse-ld=lld -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,now -Wl,-z,nocopyreloc -fstack-protector-strong -fstack-clash-protection -Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin -Wl,-rpath-link,/usr/lib ../../../../../build/pure_virtual/libpure_virtual.a /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libipcclientcerts_static.a -Wl,--version-script,libipcclientcerts.so.symbols does it crash at all or not?

@dememax
Copy link

dememax commented Apr 22, 2024

We do a lot of nose. Maybe it will better to switch in PM?
Yes, I run it manually.

@alexey-bataev
Copy link
Member

We do a lot of nose. Maybe it will better to switch in PM? Yes, I run it manually.

Ok, where do you want to discuss it?

@dememax
Copy link

dememax commented Apr 25, 2024

Status update on our experimentations with Alex.

After a little investigation, we found the problematic command (each option is on its own line for visibility):

/usr/lib/llvm/17/bin/x86_64-pc-linux-gnu-clang++-17
-U_FORTIFY_SOURCE
-D_FORTIFY_SOURCE=2
-fstack-protector-strong
-fstack-clash-protection
-fno-sized-deallocation
-fno-aligned-new
-march=native
-fno-exceptions
-fPIC
-fno-rtti
-ffunction-sections
-fdata-sections
-fno-exceptions
-fno-math-errno
-pthread
-O3
-fomit-frame-pointer
-funwind-tables
-shared
-Wl,-z,defs
-Wl,--gc-sections
-Wl,-h,libxul.so
-o libxul.so
/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/toolkit/library/build/libxul_so.list
-B
/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/build/unix/elfhack
-Wl,--real-linker,/usr/lib/llvm/17/bin/ld.lld
-flto=thin
-Wl,-plugin-opt=-import-instr-limit=10
-Wl,-plugin-opt=-import-hot-multiplier=30
-lpthread
-Wl,-O1
-Wl,--as-needed
-Wl,--undefined-version
-Wl,--compress-debug-sections=zlib
-Wl,-rpath=/usr/lib64/firefox,--enable-new-dtags
-fuse-ld=lld
-Wl,-z,noexecstack
-Wl,-z,text
-Wl,-z,relro
-Wl,-z,now
-Wl,-z,nocopyreloc
-fstack-protector-strong
-fstack-clash-protection
-Wl,-rpath-link,/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/dist/bin
-Wl,-rpath-link,/usr/lib
../../../js/src/build/libjs_static.a
../../../build/pure_virtual/libpure_virtual.a
/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libgkrust.a
../../../security/sandbox/linux/libmozsandbox.so
../../../config/external/gkcodecs/libgkcodecs.so
../../../config/external/lgpllibs/liblgpllibs.so
../../../config/external/sqlite/libmozsqlite3.so
../../../widget/gtk/mozgtk/libmozgtk.so
../../../widget/gtk/mozwayland/libmozwayland.so
-Wl,--version-script,libxul.so.symbols
-licui18n
-licuuc
-licudata
-lresolv
-laom
-ldav1d
-lasound
-lrt
-lX11
-lXcomposite
-lXdamage
-lXext
-lXfixes
-lXrandr
-lXrender
-lpthread
-ldl
-lc
-lffi
-lplds4
-lplc4
-lnspr4
-lz
-lm
-lssl3
-lsmime3
-lnss3
-lnssutil3
-lfreetype
-lfontconfig
-lgdk-3
-lharfbuzz
-lpangocairo-1.0
-lpango-1.0
-lgtk-3
-latk-1.0
-lcairo
-lcairo-gobject
-lgdk_pixbuf-2.0
-lglib-2.0
-lgobject-2.0
-lgio-2.0
-lgraphite2
-lpng16
-lwebp
-lwebpdemux
-levent
-lvpx
-lpixman-1
-ldbus-1
-lxcb-shm
-lX11-xcb
-lxcb
-lXcursor
-lXi

Alex proposed to add these options to the command:

-Wl,-plugin-opt=-print-on-crash
-Wl,-plugin-opt=-print-module-scope
-Wl,-plugin-opt=-print-on-crash-path=repro.ll

The crash was reproduced, but unfortunately, the repro.ll file wasn't produced. The output was:

ld.lld: warning: Linking two modules of different target triples: '/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/x86_64-unknown-linux-gnu/release/libgkrust.a(nsstring-e605d9a4b5ee90ce.nsstring.a86a7024c2d63334-cgu.0.rcgu.o at 155447734)' is 'x86_64-unknown-linux-gnu' whereas '/var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/toolkit/library/build/../../../netwerk/protocol/http/Unified_cpp_protocol_http4.o' is 'x86_64-pc-linux-gnu'

LLVM ERROR: too many signal callbacks already registered
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
 #0 0x00007f5db61b02d3 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xdb02d3)
 #1 0x00007f5db61adf14 llvm::sys::RunSignalHandlers() (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xdadf14)
 #2 0x00007f5db61ae096 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xdae096)
 #3 0x00007f5db4e6fca0 (/lib64/libc.so.6+0x39ca0)
 #4 0x00007f5db4ec3e3c (/lib64/libc.so.6+0x8de3c)
 #5 0x00007f5db4e6fbf6 gsignal (/lib64/libc.so.6+0x39bf6)
 #6 0x00007f5db4e588b8 abort (/lib64/libc.so.6+0x228b8)
 #7 0x00007f5db5c02570 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x802570)
 #8 0x00007f5db60e61cc llvm::report_fatal_error(char const*, bool) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xce61cc)
 #9 0x00007f5db61ae753 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xdae753)
#10 0x00007f5db9d0d0be llvm::PrintCrashIRInstrumentation::registerCallbacks(llvm::PassInstrumentationCallbacks&) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x490d0be)
#11 0x00007f5db9d12821 llvm::StandardInstrumentations::registerCallbacks(llvm::PassInstrumentationCallbacks&, llvm::AnalysisManager<llvm::Module>*) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x4912821)
#12 0x00007f5db7c2029f (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x282029f)
#13 0x00007f5db7c226a2 llvm::lto::opt(llvm::lto::Config const&, llvm::TargetMachine*, unsigned int, llvm::Module&, bool, llvm::ModuleSummaryIndex*, llvm::ModuleSummaryIndex const*, std::vector<unsigned char, std::allocator<unsigned char>> const&) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x28226a2)
#14 0x00007f5db7c23f8e (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x2823f8e)
#15 0x00007f5db7c24dcd llvm::lto::thinBackend(llvm::lto::Config const&, unsigned int, std::function<llvm::Expected<std::unique_ptr<llvm::CachedFileStream, std::default_delete<llvm::CachedFileStream>>> (unsigned int, llvm::Twine const&)>, llvm::Module&, llvm::ModuleSummaryIndex const&, llvm::StringMap<std::unordered_set<unsigned long, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<unsigned long>>, llvm::MallocAllocator> const&, llvm::DenseMap<unsigned long, llvm::GlobalValueSummary*, llvm::DenseMapInfo<unsigned long, void>, llvm::detail::DenseMapPair<unsigned long, llvm::GlobalValueSummary*>> const&, llvm::MapVector<llvm::StringRef, llvm::BitcodeModule, llvm::DenseMap<llvm::StringRef, unsigned int, llvm::DenseMapInfo<llvm::StringRef, void>, llvm::detail::DenseMapPair<llvm::StringRef, unsigned int>>, llvm::SmallVector<std::pair<llvm::StringRef, llvm::BitcodeModule>, 0u>>*, std::vector<unsigned char, std::allocator<unsigned char>> const&) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x2824dcd)
#16 0x00007f5db7c0a906 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x280a906)
#17 0x00007f5db7c17713 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0x2817713)
#18 0x00007f5db60a73e4 (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xca73e4)
#19 0x00007f5db6144c66 llvm::ThreadPool::processTasks(llvm::ThreadPoolTaskGroup*) (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xd44c66)
#20 0x00007f5db61452dd (/usr/lib/llvm/17/bin/../lib64/libLLVM-17.so+0xd452dd)
#21 0x00007f5db4ec2080 (/lib64/libc.so.6+0x8c080)
#22 0x00007f5db4f372ac (/lib64/libc.so.6+0x1012ac)
Failed to hack libxul.so: basic_ios::clear: iostream error
x86_64-pc-linux-gnu-clang++-17: error: linker command failed with exit code 1 (use -v to see invocation)

maxmsi /var/tmp/portage/www-client/firefox-125.0.1/work/firefox_build/toolkit/library/build # echo $?
1

Next, Alex proposed to use other options instead of -plugin-opt=-print-*:

-Wl,-plugin-opt=save-temps
-Wl,-plugin-opt=-opt-bisect-limit=-1

The crash was reproduced, but it wasn't enough to analyze the source of the problem.
Then, we tried to see if we could reproduce the crash with different values of -plugin-opt=-opt-bisect-limit option.
There was a hypothesis that the command starts to fail at some point, and we could find this value.
But the command fails with different values of the limit, and not in a monotonous way.

@torvic9
Copy link

torvic9 commented May 6, 2024

I can confirm the same (or very similar) issue when compiling Firefox 125.x with PGO and LTO.

It happens both with clang17 and clang18 running an up-to-date Arch Linux installation.

The error seems to happen when compiling (not linking) the Firefox source file dom/canvas/DrawTargetWebGl.cpp [1] .

I tried passing -fno-tree-vectorize -fno-slp-vectorize to the whole dom/canvas source files, which made the build proceed a little further, but ultimately still crashing at one of the last steps.

I haven't tried yet passing these parameters to the whole build and currently I do not have access to my build machine, so for the moment I can't provide the exact info.

My CFLAGS are the default Arch CFLAGS [2] except for -march=x86-64-v3 instead of the default -march/-mcpu flags.


[1] https://hg.mozilla.org/releases/mozilla-release/file/tip/dom/canvas/DrawTargetWebgl.cpp
[2] https://gitlab.archlinux.org/archlinux/packaging/packages/pacman/-/blob/main/makepkg.conf

@alexey-bataev
Copy link
Member

I can confirm the same (or very similar) issue when compiling Firefox 125.x with PGO and LTO.

It happens both with clang17 and clang18 running an up-to-date Arch Linux installation.

The error seems to happen when compiling (not linking) the Firefox source file dom/canvas/DrawTargetWebGl.cpp [1] .

I tried passing -fno-tree-vectorize -fno-slp-vectorize to the whole dom/canvas source files, which made the build proceed a little further, but ultimately still crashing at one of the last steps.

I haven't tried yet passing these parameters to the whole build and currently I do not have access to my build machine, so for the moment I can't provide the exact info.

My CFLAGS are the default Arch CFLAGS [2] except for -march=x86-64-v3 instead of the default -march/-mcpu flags.

[1] https://hg.mozilla.org/releases/mozilla-release/file/tip/dom/canvas/DrawTargetWebgl.cpp [2] https://gitlab.archlinux.org/archlinux/packaging/packages/pacman/-/blob/main/makepkg.conf

If the compiler crashes at source file, it should save the shell script and the preprocessed source file. The source file itself does not help, need the preprocessed output, generated by the comnpiler

@alexey-bataev
Copy link
Member

Finally found the reproducer and was able to reproduce the crash with 17.0.4 and 18.1.5. But the crash does not reproduce with the trunk, looks like the issue is fixed already and it will be fixed in llvm 19.0

@torvic9
Copy link

torvic9 commented May 6, 2024

The issue does NOT appear if I append

-fno-tree-vectorize -fno-slp-vectorize

to the CFLAGS.

Build finishes normally without errors.

@alexey-bataev, thanks for the feedback, maybe we can find out what commit fixed this issue and have it backported to the still active 18.x branch?

EDIT: It also needs -march=x86-64, x86-64-v3 doesn't work either.
Also, I couldn't get Cross+FullLTO to work but that is a different issue.

@alexey-bataev
Copy link
Member

The issue does NOT appear if I append

-fno-tree-vectorize -fno-slp-vectorize

to the CFLAGS.

Build finishes normally without errors.

@alexey-bataev, thanks for the feedback, maybe we can find out what commit fixed this issue and have it backported to the still active 18.x branch?

I tried to identify the fix, but there are too many potential candidates with too many dependencies. I can try to prepare a fix but it won't be a backport, it will be completely separate patch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
crash Prefer [crash-on-valid] or [crash-on-invalid] llvm:SLPVectorizer
Projects
None yet
Development

No branches or pull requests

6 participants