cargo build release extremely slow.

[light4]

I tried this code:

https://github.com/light4/databend/tree/split_common_functions

Build with command cargo build -Ztimings -p common-functions-scalars --release -vvv

Build before this commit is fine, https://github.com/light4/databend/commit/d6bad0a7964042ea3c5d0452ce7f9d04cb82b990

I expected to see this happen: build time should be almost the same, at least in less 5min.

Instead, this happened: It will take about 18min to finish the build.

If I remove codegen-units = 1in this line, it compiles in about 5min.
https://github.com/light4/databend/blob/split_common_functions/Cargo.toml#L49

Meta

rustc --version --verbose:

rustc 1.61.0-nightly (8769f4ef2 2022-03-02)
binary: rustc
commit-hash: 8769f4ef2fe1efddd1f072485f97f568e7328f79
commit-date: 2022-03-02
host: x86_64-unknown-linux-gnu
release: 1.61.0-nightly
LLVM version: 14.0.0

Process thread info:

❯ ps -Tl -p 316214                                                                                                                                                            
F S   UID     PID    SPID    PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD                                                                                             0 S  1000  316214  316214  307619  0  80   0 - 2091266 futex_ pts/5  00:00:00 rustc
1 S  1000  316214  316215  307619 30  80   0 - 2091266 futex_ pts/5  00:00:58 rustc
1 S  1000  316214  316227  307619  0  80   0 - 2091266 futex_ pts/5  00:00:00 rustc             
1 S  1000  316214  316228  307619  0  80   0 - 2091266 futex_ pts/5  00:00:00 rustc
1 R  1000  316214  316233  307619 99  80   0 - 2091266 -    pts/5    00:02:13 opt common_func              

GDB `thread apply all bt`

Thread 5 (Thread 0x7faf677ff640 (LWP 316233) "opt common_func"):
#0  0x00007fb0e68464f7 in llvm::GlobalsAAResult::DeletionCallbackHandle::deleted() () from /home/light4/.rustup/toolchains/nightly-2022-03-03-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-14-rust-1.61.0-nightly.so
#1  0x00007fb0e587e31e in llvm::GlobalObject::~GlobalObject() () from /home/light4/.rustup/toolchains/nightly-2022-03-03-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-14-rust-1.61.0-nightly.so
#2  0x00007fb0e53a1491 in llvm::Function::~Function() () from /home/light4/.rustup/toolchains/nightly-2022-03-03-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-14-rust-1.61.0-nightly.so
#3  0x00007fb0e539eb08 in llvm::InlinerPass::run(llvm::LazyCallGraph::SCC&, llvm::AnalysisManager<llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&>&, llvm::LazyCallGraph&, llvm::CGSCCUpdateResult&) () from /home/light4/.rustup/toolchains/nightly-2022-03-03-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-14-rust-1.61.0-nightly.so
#4  0x00007fb0e539d12d in llvm::detail::PassModel<llvm::LazyCallGraph::SCC, llvm::InlinerPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&>, llvm::LazyCallGraph&, llvm::CGSCCUpdateResult&>::run(llvm::LazyCallGraph::SCC&, llvm::AnalysisManager<llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&>&, llvm::LazyCallGraph&, llvm::CGSCCUpdateResult&) () from /home/light4/.rustup/toolchains/nightly-2022-03-03-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-14-rust-1.61.0-nightly.so
#5  0x00007fb0e57e8fc6 in llvm::PassManager<llvm::LazyCallGraph::SCC, llvm::AnalysisManager<llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&>, llvm::LazyCallGraph&, llvm::CGSCCUpdateResult&>::run(llvm::LazyCallGraph::SCC&, llvm::AnalysisManager<llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&>&, llvm::LazyCallGraph&, llvm::CGSCCUpdateResult&) () from /home/light4/.rustup/toolchains/nightly-2022-03-03-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-14-rust-1.61.0-nightly.so
#6  0x00007fb0e57e8bfd in llvm::detail::PassModel<llvm::LazyCallGraph::SCC, llvm::PassManager<llvm::LazyCallGraph::SCC, llvm::AnalysisManager<llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&>, llvm::LazyCallGraph&, llvm::CGSCCUpdateResult&>, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&>, llvm::LazyCallGraph&, llvm::CGSCCUpdateResult&>::run(llvm::LazyCallGraph::SCC&, llvm::AnalysisManager<llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&>&, llvm::LazyCallGraph&, llvm::CGSCCUpdateResult&) () from /home/light4/.rustup/toolchains/nightly-2022-03-03-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-14-rust-1.61.0-nightly.so
#7  0x00007fb0e57e3dba in llvm::DevirtSCCRepeatedPass::run(llvm::LazyCallGraph::SCC&, llvm::AnalysisManager<llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&>&, llvm::LazyCallGraph&, llvm::CGSCCUpdateResult&) () from /home/light4/.rustup/toolchains/nightly-2022-03-03-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-14-rust-1.61.0-nightly.so
#8  0x00007fb0e57e3b6d in llvm::detail::PassModel<llvm::LazyCallGraph::SCC, llvm::DevirtSCCRepeatedPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&>, llvm::LazyCallGraph&, llvm::CGSCCUpdateResult&>::run(llvm::LazyCallGraph::SCC&, llvm::AnalysisManager<llvm::LazyCallGraph::SCC, llvm::LazyCallGraph&>&, llvm::LazyCallGraph&, llvm::CGSCCUpdateResult&) () from /home/light4/.rustup/toolchains/nightly-2022-03-03-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-14-rust-1.61.0-nightly.so
#9  0x00007fb0e55790a5 in llvm::ModuleToPostOrderCGSCCPassAdaptor::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) () from /home/light4/.rustup/toolchains/nightly-2022-03-03-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-14-rust-1.61.0-nightly.so
#10 0x00007fb0e55786fd in llvm::detail::PassModel<llvm::Module, llvm::ModuleToPostOrderCGSCCPassAdaptor, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) () from /home/light4/.rustup/toolchains/nightly-2022-03-03-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-14-rust-1.61.0-nightly.so
#11 0x00007fb0e5bcfc24 in llvm::PassManager<llvm::Module, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) () from /home/light4/.rustup/toolchains/nightly-2022-03-03-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-14-rust-1.61.0-nightly.so
#12 0x00007fb0e5bcf2b5 in llvm::ModuleInlinerWrapperPass::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) () from /home/light4/.rustup/toolchains/nightly-2022-03-03-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-14-rust-1.61.0-nightly.so
#13 0x00007fb0e5bcf11d in llvm::detail::PassModel<llvm::Module, llvm::ModuleInlinerWrapperPass, llvm::PreservedAnalyses, llvm::AnalysisManager<llvm::Module>>::run(llvm::Module&, llvm::AnalysisManager<llvm::Module>&) () from /home/light4/.rustup/toolchains/nightly-2022-03-03-x86_64-unknown-linux-gnu/bin/../lib/../lib/libLLVM-14-rust-1.61.0-nightly--Type <RET> for more, q to quit, c to continue without paging--Quit

Full cargo timing image

Maybe related to #93774 ?

[light4]

Maybe it's normal.

codegen-units	build time
16 (default)	158.5s (2m 38.5s)
8	260.4s (4m 20.4s)
4	431.2s (7m 11.2s)
2	722.2s (12m 2.2s)
1	982.2s (16m 22.2s)

Reducing codegen-units will increase your compile time by a lot.