Compiling attached file which uses boost::container::flat_set is very slow with clang & create large .text section

[DominiquePelle-TomTom]

Compiling the attached file brands.cpp file (or its preprocessed
file brands-preproc.cpp) which uses boost::container::flat_set is
very slow with Clang (any version) when using Boost 1.81.0.
It was much faster with Boost 1.80.0 somehow and is much faster
with any version of GCC.

Furthermore, the size of the .text section is much bigger
when using Boost 1.81.0 than with Boost 1.80.0 and when
using Clang (any version). The .text section is also much smaller
with any version of GCC.

The source file brands.cpp is a simplification of an actual file in my
project. Note that the code is in anonymous namespace (removing the
anonymous namespace makes the compilation time much faster,
presumably because there are more optimizations in the
anonymous namespace).

Attached file: brands.cpp.gz

Attached preprocessed source file (with Boost 1.81):
brands-preproc.cpp.gz

Benchmarks with Clang 16 + Boost 1.81.0 (on Ubuntu 22.04):

$ time clang++-16 -isystem boost-1.81 -O2 -std=c++14 -o brands.o -c brands.cpp

real    2m3.055s
user    2m2.411s
sys     0m0.632s

$ size brands.o
   text    data     bss     dec     hex filename
2635353   20032      48 2655433  2884c9 brands.o

Compare with GCC 12.1:

$ time gcc-12 -isystem boost-1.81 -O2 -std=c++14 -o brands.o -c brands.cpp

real    0m2.582s
user    0m2.476s
sys     0m0.105s

$ size brands.o
   text    data     bss     dec     hex filename
 182851   15088      48  197987   30563 brands.o

In other words:

• compilation is 47.6 times slower with Clang 16 than with GCC 12.1
• .text section is 14.1 times bigger with Clang 16 than with GCC 12.1

It's even worse with older Clang compilers. The table below shows
the compilation time of attached brands.cpp file and size of the .text
section with various compilers using Boost 1.80 and Boost 1.81:

compiler   boost   compilation time    size .text section
--------   -----   ----------------    ------------------
clang-16   1.81         125.582 sec       2,635,353 bytes
clang-16   1.80           2.164 sec         101,280 bytes

clang-15   1.81         137.508 sec       2,861,332 bytes
clang-15   1.80           2.527 sec         107,944 bytes

clang-14   1.81         153.504 sec       3,033,487 bytes
clang-14   1.80           2.294 sec         111,599 bytes

clang-5.0  1.81         276.031 sec       2,720,187 bytes
clang-5.0  1.80           2.328 sec         126,811 bytes

gcc-12.1   1.81           2.821 sec         139,833 bytes
gcc-12.1   1.80           2.505 sec         140,217 bytes

gcc-11.3   1.81           2.128 sec         182,038 bytes
gcc-11.3   1.80           2.121 sec         182,158 bytes

gcc-7.5    1.81           1.943 sec         121,497 bytes
gcc-7.5    1.80           1.707 sec         122,050 bytes

Full versions of compiler used in above table (--version option):

• Ubuntu clang version 16.0.2 (++20230414073413+b5aa566a7e53-1~exp1~20230414073428.73)
• Ubuntu clang version 15.0.7
• Ubuntu clang version 14.0.0-1ubuntu1
• clang version 5.0.2 (tags/RELEASE_502/final)
• gcc-12 (Ubuntu 12.1.0-2ubuntu1~22.04) 12.1.0
• gcc-11 (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
• gcc-7 (Ubuntu 7.5.0-6ubuntu2) 7.5.0

I also reported this issue for Boost since there is a large
compilation time & .text section regression from Boost 1.80
to Boost 1.81. See:
https://marc.info/?l=boost-users&m=168173836211082&w=2

[shafik]

CC @schittir Erich thought you might be able to figure what is going on here

[schittir]

Hi all, sharing some updates here:

1. Inlining causes compile time slowness:

time clang++ -O2 -isystem ../../boost_1_81_0 -std=c++14 -o brands.o -c brands.cpp

real    2m32.632s
user    2m31.626s
sys     0m0.976s

time  clang++ -O2 -fno-inline -isystem ../../boost_1_81_0 -std=c++14 -o brands.o -c brands.cpp

real    0m1.102s
user    0m1.086s
sys     0m0.017s

size brands.o
   text    data     bss     dec     hex filename
 110678   20136      48  130862   1ff2e brands.o


clang version 17.0.0 

A snapshot of clang's hotspots profile is here:

Further work is required on narrowing down inlining heuristics/decisions responsible for this.

Additional data:

time clang++ -O1 -isystem ../../boost_1_81_0 -std=c++14 -o brands.o -c brands.cpp

real    1m9.974s
user    1m9.532s
sys     0m0.384s
size brands.o
   text    data     bss     dec     hex filename
1616252   20072      48 1636372  18f814 brands.o

time clang++ -O0 -isystem ../../boost_1_81_0 -std=c++14 -o brands.o -c brands.cpp

real    0m0.698s
user    0m0.595s
sys     0m0.012s
size brands.o
   text    data     bss     dec     hex filename
 465996   20136      48  486180   76b24 brands.o

[schittir]

time clang++ -O2 -isystem ../../boost_1_81_0 -std=c++14 -o brands.o -c brands.cpp  -ftime-report
===-------------------------------------------------------------------------===
                          Pass execution timing report
===-------------------------------------------------------------------------===
  Total Execution Time: 35.8005 seconds (35.8020 wall clock)

   ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
   8.1035 ( 22.8%)   0.0069 (  2.6%)   8.1105 ( 22.7%)   8.1116 ( 22.7%)  InstCombinePass
   6.9403 ( 19.5%)   0.0000 (  0.0%)   6.9403 ( 19.4%)   6.9414 ( 19.4%)  SLPVectorizerPass
   3.0727 (  8.6%)   0.0240 (  8.9%)   3.0967 (  8.6%)   3.0971 (  8.7%)  LoopUnrollPass
   3.0392 (  8.6%)   0.0080 (  2.9%)   3.0472 (  8.5%)   3.0475 (  8.5%)  LoopVectorizePass
   2.8054 (  7.9%)   0.0360 ( 13.3%)   2.8414 (  7.9%)   2.8417 (  7.9%)  LoopLoadEliminationPass
   1.9406 (  5.5%)   0.0367 ( 13.5%)   1.9773 (  5.5%)   1.9775 (  5.5%)  GVNPass
   1.1034 (  3.1%)   0.0362 ( 13.4%)   1.1396 (  3.2%)   1.1391 (  3.2%)  IndVarSimplifyPass
   1.0852 (  3.1%)   0.0263 (  9.7%)   1.1115 (  3.1%)   1.1107 (  3.1%)  LoopIdiomRecognizePass
   1.0223 (  2.9%)   0.0010 (  0.4%)   1.0233 (  2.9%)   1.0234 (  2.9%)  SimplifyCFGPass
   0.9124 (  2.6%)   0.0086 (  3.2%)   0.9210 (  2.6%)   0.9210 (  2.6%)  JumpThreadingPass
   0.8717 (  2.5%)   0.0167 (  6.2%)   0.8884 (  2.5%)   0.8878 (  2.5%)  LoopDeletionPass
   0.6329 (  1.8%)   0.0393 ( 14.5%)   0.6722 (  1.9%)   0.6722 (  1.9%)  InlinerPass
   0.5591 (  1.6%)   0.0051 (  1.9%)   0.5642 (  1.6%)   0.5643 (  1.6%)  CorrelatedValuePropagationPass
   0.5301 (  1.5%)   0.0005 (  0.2%)   0.5306 (  1.5%)   0.5307 (  1.5%)  LoopSimplifyPass
   0.4765 (  1.3%)   0.0046 (  1.7%)   0.4811 (  1.3%)   0.4809 (  1.3%)  LoopFullUnrollPass
   0.2949 (  0.8%)   0.0104 (  3.8%)   0.3053 (  0.9%)   0.3049 (  0.9%)  LICMPass


                         Miscellaneous Ungrouped Timers
===-------------------------------------------------------------------------===

   ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
  151.7307 (100.0%)   1.8598 ( 99.8%)  153.5904 (100.0%)  153.6130 (100.0%)  Code Generation Time
   0.0613 (  0.0%)   0.0041 (  0.2%)   0.0655 (  0.0%)   0.0657 (  0.0%)  LLVM IR Generation Time
  151.7920 (100.0%)   1.8639 (100.0%)  153.6559 (100.0%)  153.6787 (100.0%)  Total


===-------------------------------------------------------------------------===
                          Pass execution timing report
===-------------------------------------------------------------------------===
  Total Execution Time: 117.1235 seconds (117.1344 wall clock)

   ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
  **61.7840 ( 53.5%)   0.2200 ( 13.9%)  62.0040 ( 52.9%)  62.0116 ( 52.9%)  Branch Probability Basic Block Placement**
   9.4776 (  8.2%)   0.1840 ( 11.7%)   9.6615 (  8.2%)   9.6635 (  8.2%)  Greedy Register Allocator #2
   9.2535 (  8.0%)   0.0341 (  2.2%)   9.2876 (  7.9%)   9.2856 (  7.9%)  Loop Strength Reduction
   5.1350 (  4.4%)   0.4159 ( 26.3%)   5.5510 (  4.7%)   5.5520 (  4.7%)  X86 DAG->DAG Instruction Selection
   4.8536 (  4.2%)   0.0000 (  0.0%)   4.8536 (  4.1%)   4.8543 (  4.1%)  CodeGen Prepare
   3.4199 (  3.0%)   0.0002 (  0.0%)   3.4201 (  2.9%)   3.4206 (  2.9%)  Machine Common Subexpression Elimination
   2.9658 (  2.6%)   0.0375 (  2.4%)   3.0033 (  2.6%)   3.0016 (  2.6%)  Canonicalize Freeze Instructions in Loops
   2.5064 (  2.2%)   0.0081 (  0.5%)   2.5145 (  2.1%)   2.5152 (  2.1%)  Control Flow Optimizer
   2.4736 (  2.1%)   0.0167 (  1.1%)   2.4903 (  2.1%)   2.4909 (  2.1%)  Simple Register Coalescing
   1.3547 (  1.2%)   0.0001 (  0.0%)   1.3548 (  1.2%)   1.3550 (  1.2%)  Machine Cycle Info Analysis
   1.1887 (  1.0%)   0.0069 (  0.4%)   1.1956 (  1.0%)   1.1952 (  1.0%)  Induction Variable Users
   0.5385 (  0.5%)   0.4482 ( 28.4%)   0.9867 (  0.8%)   0.9869 (  0.8%)  X86 Assembly Printer