Support c/c++ library internal LTO

[NobodyXu]

In #1564 we disabled LTO if linker-plugin-lto is not passed in to fix compilation on certain environment.

We could reclaim back the performance, by doing a c/c++ library internal LTO, similar to rustc's crate internal LTO.

To find these tools, we might want 'clang -print-resource-dir', or combined with env var lookup LLVM_INSTALL_DIR

We should also allow build script to override lto mode.

Did some research, for clang it seems to support this

llvm-link file1.bc file2.bc file3.bc -o merged.bc
llc merged.bc -filetype=obj -o merged.o
I didn't find anyway to achieve that for gcc.

Ok so the opt command from llvm can actually do the optimization, might worth trying to see if it works.

We could do

llvm-link file1.bc file2.bc file3.bc -o merged.bc
opt -O{level} merged.bc
llc merged.bc -filetype=obj -o merged.o

Originally posted by @NobodyXu in #1463

[NobodyXu]

Seems that we can use clang -O3 -c merged.bc to do the optimization and codegen, that'd makes things much easier

[dot-asm]

Seems that we can use clang -O3 merged.bc to do the optimization and codegen, that'd makes things much easier

I don't follow. I thought that the goal was to produce an .a that is IPO-ed, or "thin-lto-ed" in Rust terms. clang -O3 will attempt to link a final application.

[NobodyXu]

Thanks I forgot to put -c in it, I need to find some time to experiment it on my PC and see if it can do the job

[dot-asm]

I would advocate in favour of implementing this as a feature and even make it independent of Rust -> clang flag mappings. Indeed, it would more than natural to give crate maintainers a way to perform localized IPO independent of user setting environment variables. Just like it's more natural to configure Rust lto through crate's Cargo.toml.

[NobodyXu]

So tested this out and we just need llvm-link:

clang -O3 -flto -c a.c b.c
llvm-link a.o b.o -o merged.o
clang -O3 -x ir -c merged.o -o final.o

We could use ar_archive_writer instead of llvm-link but it'd be more complicated, plus we will pull in more dependenciesz

[dot-asm]

And if you throw in -flto -ffat-lto-objects(*) to the last command the result would be IPO-ed object module with embedded bitcode which should allow end user to opt for -Clinker-plugin-lto at their discretion. I mean imagine a crate maintainer who wants to apply IPO to their C-based crate for everybody, but leave the door open for crazies out there.

(*) To be conditional on clang>=18.

[NobodyXu]

$ clang --version
Ubuntu clang version 20.1.2 (Oubuntul)
Target: x86_64-pc-Linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm-20/bin

For llvm-link, we can use an env var LLVM_LINK, or fallback to look it up in clang InstalledDir, which will guarantee to be compatible with the clang we are using.

[dot-asm]

I advocate for parsing -print-search-dirs for this kind of things. And to a significantly wider extent than it currently is, as noted in

cc-rs/src/lib.rs

Lines 3320 to 3325 in 1ca8b2a

	} else if target.arch == "wasm32" || target.arch == "wasm64" {
	// Formally speaking one should be able to use this approach,
	// parsing -print-search-dirs output, to cover all clang targets,
	// including Android SDKs and other cross-compilation scenarios...
	// And even extend it to gcc targets by searching for "ar" instead
	// of "llvm-ar"...

.

[dot-asm]

I advocate for parsing -print-search-dirs for this kind of things.

On a related note. Elsewhere in src/lib.rs it mentions that one can't use it with clang-cl. This turned to be false, it is possible to query clang-cl with /clang:-print-search-dirs command line option. In other words one can extend this approach even to Windows.

[dot-asm]

it is possible to query clang-cl with /clang:-print-search-dirs command line option

Hmmm, one can actually query it with just --print-search-dirs, i.e. with extra dash. And the extra dash works even with "normal" clang. As well as gcc(*). So one can unify on it across all targets.

(*) goes back at least to gcc 4.6 and clang 3.8, so "universality" is safe to assume.

[NobodyXu]

it is possible to query clang-cl with /clang:-print-search-dirs command line option

Hmmm, one can actually query it with just --print-search-dirs, i.e. with extra dash. And the extra dash works even with "normal" clang. As well as gcc(*). So one can unify on it across all targets.

Thanks, that's awesome, will open a PR to fix that first, and then a PR for the lto stuff

[NobodyXu]

cc @dot-asm opened #1587, would be great if you can have a look please