Profile-Guided Optimization fail to compile with segmentation fault

[whfuyn]

I'm trying profile-guided optimization, but it failed to compile with segmentation fault.

RUSTFLAGS="-Cprofile-use=/tmp/pgo-data/merged.profdata"
cargo build --release --target=x86_64-unknown-linux-gnu

Compiling kms v0.1.0 (kms_standalone)

error: could not compile `kms`

Caused by:
  process didn't exit successfully: `rustc --crate-name kms --edition=2021 ... `
  (signal: 7, SIGBUS: access to undefined memory)

Sometimes (signal: 11, SIGSEGV: invalid memory reference).

I also tried a hello-world project, and it works fine.

My project: https://github.com/cita-cloud/kms_standalone

Meta

rustc --version --verbose:

rustc 1.58.0-nightly (4e0d3973f 2021-10-27)
binary: rustc
commit-hash: 4e0d3973fafdfb1c51011bc74e44257b5e3863f1
commit-date: 2021-10-27
host: x86_64-unknown-linux-gnu
release: 1.58.0-nightly
LLVM version: 13.0.

Backtrace

RUST_BACKTRACE=1 RUSTFLAGS="-Cprofile-use=/tmp/pgo-data/merged.profdata" cargo build --release --target=x86_64-unknown-linux-gnu

   Compiling kms v0.1.0 (/home/user/kms_standalone)
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x4c7383)[0x7fe6f89cd383]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x141f0)[0x7fe6f81771f0]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.0-nightly.so(_ZNK4llvm7Mangler17getNameWithPrefixERNS_15SmallVectorImplIcEEPKNS_11GlobalValueEb+0x2e3)[0x7fe6f56b2dc3]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.0-nightly.so(_ZNK4llvm13TargetMachine9getSymbolEPKNS_11GlobalValueE+0x6f)[0x7fe6f56b233f]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.0-nightly.so(_ZNK4llvm24TargetLoweringObjectFile21emitCGProfileMetadataERNS_10MCStreamerERNS_6ModuleE+0x164)[0x7fe6f6290524]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.0-nightly.so(_ZNK4llvm27TargetLoweringObjectFileELF18emitModuleMetadataERNS_10MCStreamerERNS_6ModuleE+0xd5)[0x7fe6f628ff75]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.0-nightly.so(_ZN4llvm10AsmPrinter14doFinalizationERNS_6ModuleE+0x25d)[0x7fe6f61da04d]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.0-nightly.so(_ZN4llvm13FPPassManager14doFinalizationERNS_6ModuleE+0x2d)[0x7fe6f61f6a0d]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.0-nightly.so(_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE+0x3c5)[0x7fe6f61f60e5]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x2444ca4)[0x7fe6fa94aca4]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x243fa0f)[0x7fe6fa945a0f]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x2442600)[0x7fe6fa948600]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x23d51ab)[0x7fe6fa8db1ab]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x23cf8ed)[0x7fe6fa8d58ed]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x24038f0)[0x7fe6fa9098f0]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x24285ac)[0x7fe6fa92e5ac]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/libstd-8adcca4f1427867b.so(rust_metadata_std_fcea40badc263c8f+0xa94c3)[0x7fe6f82404c3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9450)[0x7fe6f816c450]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7fe6f807ad53]
error: could not compile `kms`

Caused by:
  process didn't exit successfully: `rustc --crate-name kms --edition=2021 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C panic=abort -C lto -C metadata=5117a81938dc7a68 -C extra-filename=-5117a81938dc7a68 --out-dir /home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/user/kms_standalone/target/release/deps --extern anyhow=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libanyhow-ba82fabb03a89ae9.rlib --extern clap=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libclap-301235565c36679a.rlib --extern efficient_sm2=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libefficient_sm2-12c48832db1cef3b.rlib --extern hex=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libhex-f5b979ff7f033c73.rlib --extern jemallocator_global=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libjemallocator_global-4fe19d1293a51495.rlib --extern libsm=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/liblibsm-9c872b881d08502d.rlib --extern lru=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/liblru-e67c713f2a010349.rlib --extern parking_lot=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libparking_lot-fa67ff07658cabc7.rlib --extern prost=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libprost-191245b8561b58f7.rlib --extern rand=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/librand-7a6361873a8c2f60.rlib --extern secrecy=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libsecrecy-d5955e43839579ac.rlib --extern serde=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libserde-eb01c2aa8aabd964.rlib --extern sqlx=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libsqlx-98d357f15c1c6a01.rlib --extern thiserror=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libthiserror-27a3e6dc89560ed9.rlib --extern tokio=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libtokio-59441228a37ce303.rlib --extern toml=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libtoml-d3591f74315998b5.rlib --extern tonic=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libtonic-8d404e44f672e7d0.rlib --extern tracing=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libtracing-dd8d5593a2e7f02b.rlib --extern tracing_appender=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libtracing_appender-90d6db31334c96ee.rlib --extern tracing_subscriber=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/libtracing_subscriber-c6dbb15fb600395e.rlib --extern url=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/deps/liburl-886159ac8fb7d14c.rlib -Cprofile-use=/tmp/pgo-data/merged.profdata -L native=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/build/jemalloc-sys-561d18810a756faf/out/build/lib -L native=/home/user/kms_standalone/target/x86_64-unknown-linux-gnu/release/build/ring-c7ba6bf9e094ac7e/out`
  (signal: 11, SIGSEGV: invalid memory reference)

[whfuyn]

I'll try to add more info later

[whfuyn]

I manage to reproduce that in a tiny example. It seems related to the LTO, and works well when lto = false.

#[tokio::main]
async fn main() {

}

[package]
name = "hello"
version = "0.1.0"
edition = "2021"

[dependencies]
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }

[profile.release]
lto = true

$ rm -rf /tmp/pgo-data
$ RUSTFLAGS="-Cprofile-generate=/tmp/pgo-data"     cargo build --release --target=x86_64-unknown-linux-gnu
   Compiling proc-macro2 v1.0.32
   Compiling unicode-xid v0.2.2
   Compiling syn v1.0.81
   Compiling libc v0.2.106
   Compiling autocfg v1.0.1
   Compiling pin-project-lite v0.2.7
   Compiling tokio v1.13.0
   Compiling quote v1.0.10
   Compiling num_cpus v1.13.0
   Compiling tokio-macros v1.5.1
   Compiling hello v0.1.0 (/home/user/hello)
    Finished release [optimized] target(s) in 9.38s

$ ./target/x86_64-unknown-linux-gnu/release/hello

$ alias rllvm-profdata=~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-profdata
$ rllvm-profdata merge -o /tmp/pgo-data/merged.profdata /tmp/pgo-data

$  RUSTFLAGS="-Cprofile-use=/tmp/pgo-data/merged.profdata" cargo build --release --target=x86_64-unknown-linux-gnu

   Compiling pin-project-lite v0.2.7
   Compiling libc v0.2.106
   Compiling num_cpus v1.13.0
   Compiling tokio v1.13.0
   Compiling hello v0.1.0 (/home/user/hello)
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x4c7383)[0x7f65e7154383]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x141f0)[0x7f65e68fe1f0]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.0-nightly.so(_ZNK4llvm7Mangler17getNameWithPrefixERNS_15SmallVectorImplIcEEPKNS_11GlobalValueEb+0x2e3)[0x7f65e3e39dc3]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.0-nightly.so(_ZNK4llvm13TargetMachine9getSymbolEPKNS_11GlobalValueE+0x6f)[0x7f65e3e3933f]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.0-nightly.so(_ZNK4llvm24TargetLoweringObjectFile21emitCGProfileMetadataERNS_10MCStreamerERNS_6ModuleE+0x164)[0x7f65e4a17524]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.0-nightly.so(_ZNK4llvm27TargetLoweringObjectFileELF18emitModuleMetadataERNS_10MCStreamerERNS_6ModuleE+0xd5)[0x7f65e4a16f75]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.0-nightly.so(_ZN4llvm10AsmPrinter14doFinalizationERNS_6ModuleE+0x25d)[0x7f65e496104d]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.0-nightly.so(_ZN4llvm13FPPassManager14doFinalizationERNS_6ModuleE+0x2d)[0x7f65e497da0d]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.0-nightly.so(_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE+0x3c5)[0x7f65e497d0e5]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x2444ca4)[0x7f65e90d1ca4]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x243fa0f)[0x7f65e90cca0f]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x2442600)[0x7f65e90cf600]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x23d51ab)[0x7f65e90621ab]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x23cf8ed)[0x7f65e905c8ed]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x24038f0)[0x7f65e90908f0]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-25ea0f19be037cbe.so(+0x24285ac)[0x7f65e90b55ac]
/home/user/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/../lib/libstd-8adcca4f1427867b.so(rust_metadata_std_fcea40badc263c8f+0xa94c3)[0x7f65e69c74c3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9450)[0x7f65e68f3450]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f65e6801d53]
error: could not compile `hello`

Caused by:
  process didn't exit successfully: `rustc --crate-name hello --edition=2021 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C lto -C metadata=f1308e41e83e897f -C extra-filename=-f1308e41e83e897f --out-dir /home/user/hello/target/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/home/user/hello/target/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/user/hello/target/release/deps --extern tokio=/home/user/hello/target/x86_64-unknown-linux-gnu/release/deps/libtokio-be8676721867c7d8.rlib -Cprofile-use=/tmp/pgo-data/merged.profdata`
  (signal: 11, SIGSEGV: invalid memory reference)

[whfuyn]

https://github.com/llvm/llvm-project/blob/e356027016c6365b3d8924f54c33e2c63d931492/llvm/lib/IR/Mangler.cpp#L74-L79

The error message points to here.

[whfuyn]

I built a stage1 rustc with config.toml:

# Includes one of the default files in src/bootstrap/defaults
profile = "codegen"
changelog-seen = 2

[llvm]
assertions = true
release-debuginfo = true

[rust]
debug = true
debug-logging = true

[build]
profiler = true

Now the error message becomes clear!

rustc: /home/user/src/rust/src/llvm-project/llvm/include/llvm/Support/Casting.h:269: 

typename llvm::cast_retty<X, Y*>::ret_type llvm::cast(Y*)
[with X = llvm::Function; Y = llvm::Value; typename llvm::cast_retty<X, Y*>::ret_type = llvm::Function*]: 

Assertion `isa<X>(Val) && "cast<Ty>() argument of incompatible type!"' failed.

Full log:

➜  hello git:(master) ✗ RUSTFLAGS="-Cprofile-use=/tmp/pgo-data/merged.profdata" cargo build --release --target=x86_64-unknown-linux-gnu
   Compiling pin-project-lite v0.2.7
   Compiling libc v0.2.106
   Compiling num_cpus v1.13.0
   Compiling tokio v1.13.0
   Compiling hello v0.1.0 (/home/user/src/hello)
rustc: /home/user/src/rust/src/llvm-project/llvm/include/llvm/Support/Casting.h:269: typename llvm::cast_retty<X, Y*>::ret_type llvm::cast(Y*) [with X = llvm::Function; Y = llvm::Value; typename llvm::cast_retty<X, Y*>::ret_type = llvm::Function*]: Assertion `isa<X>(Val) && "cast<Ty>() argument of incompatible type!"' failed.
error: could not compile `hello`

Caused by:
  process didn't exit successfully: `rustc --crate-name hello --edition=2021 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C opt-level=3 -C lto -C metadata=f7513a844da49c9b -C extra-filename=-f7513a844da49c9b --out-dir /home/user/src/hello/target/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/home/user/src/hello/target/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/user/src/hello/target/release/deps --extern tokio=/home/user/src/hello/target/x86_64-unknown-linux-gnu/release/deps/libtokio-7721661bcaabc1c1.rlib -Cprofile-use=/tmp/pgo-data/merged.profdata` (signal: 6, SIGABRT: process abort signal)

[s-knibbs]

I've also hit this issue when LTO is enabled with the following rustc version:

rustc 1.58.1 (db9d1b20b 2022-01-20)
binary: rustc
commit-hash: db9d1b20bba1968c1ec1fc49616d4742c1725b4b
commit-date: 2022-01-20
host: x86_64-unknown-linux-gnu
release: 1.58.1
LLVM version: 13.0.0

Backtrace:

/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-f63adaab7e27a794.so(+0x4dd233)[0x7f8b261e4233]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f8b259713c0]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.1-stable.so(_ZNK4llvm7Mangler17getNameWithPrefixERNS_15SmallVectorImplIcEEPKNS_11GlobalValueEb+0x2e3)[0x7f8b22f466d3]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.1-stable.so(_ZNK4llvm13TargetMachine9getSymbolEPKNS_11GlobalValueE+0xd1)[0x7f8b22f45ca1]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.1-stable.so(_ZNK4llvm24TargetLoweringObjectFile21emitCGProfileMetadataERNS_10MCStreamerERNS_6ModuleE+0x164)[0x7f8b23a35564]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.1-stable.so(_ZNK4llvm27TargetLoweringObjectFileELF18emitModuleMetadataERNS_10MCStreamerERNS_6ModuleE+0xd5)[0x7f8b23a34fb5]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.1-stable.so(_ZN4llvm10AsmPrinter14doFinalizationERNS_6ModuleE+0x25d)[0x7f8b2398d53d]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.1-stable.so(_ZN4llvm13FPPassManager14doFinalizationERNS_6ModuleE+0x2d)[0x7f8b237ce2bd]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-13-rust-1.58.1-stable.so(_ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE+0x333)[0x7f8b237cc6c3]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-f63adaab7e27a794.so(+0x216dfd4)[0x7f8b27e74fd4]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-f63adaab7e27a794.so(+0x2155e77)[0x7f8b27e5ce77]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-f63adaab7e27a794.so(+0x2159159)[0x7f8b27e60159]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-f63adaab7e27a794.so(+0x216302b)[0x7f8b27e6a02b]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-f63adaab7e27a794.so(+0x215d047)[0x7f8b27e64047]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-f63adaab7e27a794.so(+0x210ce23)[0x7f8b27e13e23]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-f63adaab7e27a794.so(+0x2141f73)[0x7f8b27e48f73]
/home/sknibbs/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/libstd-4c74cbab78ec4891.so(rust_metadata_std_b2a2bae5f6ca25d7+0xaafc3)[0x7f8b25a41fc3]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7f8b25965609]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7f8b25852293]

[zertyz]

+1 here.
I'm experiencing the exact same problem on Rust 1.59.0 (default Archlinux package) and I just confirmed that using LTO="off" allows the PGO compilation -- when LTO="fat" produces the error messages on the OP and LTO="thin" produces other error messages.

Maybe this is a bug -- or, if it is by design, a better error message is needed.

If PGO + FAT LTO is possible, I wonder what would be the combined power of those optimizations, as on my tests I found out that PGO offers a 12% improvement over "LTO off" while "LTO fat" offers 4% improvement over "LTO off"

The error messages I got when trying LTO="thin" are:

$ RUSTFLAGS="-C target-cpu=native -C profile-use=${PGO_DATA}/merged.profdata -Cllvm-args=-pgo-warn-missing-function" cargo build --release --features="tcmalloc_allocator"
(... regular compilation warnings ...)
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
warning: `ogre-datasets-converter-demo` (bin "ogre-datasets-converter-demo") generated 18 warnings
error: could not compile `ogre-datasets-converter-demo`; 18 warnings emitted

[insilications]

+1 here too.

To avoid these errors, I've build the standard libraries with "-Clinker-plugin-lto=on -Ccodegen-units=1", so that all .rlib files contain the LLVM IR bitcode instead of the native object. I placed them all inside /usr/lib64/rustlib/x86_64-unknown-linux-gnu/lib/ and remove the previous ones. Following the guide (https://doc.rust-lang.org/nightly/rustc/linker-plugin-lto.html), you can them have all the benefits of LTO along with cross LTO, for all your program and all dependencies, including over the standard library as well.

[zamazan4ik]

I guess I got the same error:

...
Compiling oxipng v5.0.1 (/Users/zamazan4ik/open_source/oxipng)
module flag identifiers must be unique (or of 'require' type)
!"CG Profile"
LLVM ERROR: Broken module found, compilation aborted!
error: could not compile `oxipng`

I got this error with Thin LTO + PGO. Switching to Fat LTO + PGO works fine.

Rust version: 1.60