Segfault during BOLT profile collection in `test_functools` (`fib`) when building with `--enable-bolt --enable-optimizations`

[ashm-dev]

Crash report

What happened?

Environment

• OS: Linux (x86_64)
• CPython: main branch
• Compiler: Clang / Clang++ (with lld)
• BOLT version: ca2b52862de907b4e4dca1d064098a28ab8269be

Steps to Reproduce

1. Configure CPython:

   CC=clang CXX=clang++ LDFLAGS='-fuse-ld=lld' ./configure --enable-bolt --enable-optimizations

2. Run build:

   make

Expected Behavior

Successful build and optimization of Python binary using BOLT.

Actual Behavior

Segmentation fault occurs while running instrumented binary to collect profile data (./python -m test --pgo --timeout=) during test_functools. The crash happens in the recursive fib function inside the BOLT runtime instrumentation layer.

Crash Log / Backtrace

Details

BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: ca2b52862de907b4e4dca1d064098a28ab8269be
BOLT-INFO: first alloc address is 0x200000
BOLT-INFO: creating new program header table at address 0x800000, offset 0x600000
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling lite mode
BOLT-INFO: 0 out of 9012 functions in the binary (0.0%) have non-empty execution profile
BOLT-INFO: validate-mem-refs updated 2 object references
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 1949
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 9003
BOLT-INSTRUMENTER: Number of function descriptors: 9003
BOLT-INSTRUMENTER: Number of branch counters: 104776
BOLT-INSTRUMENTER: Number of ST leaf node counters: 52724
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 157500
BOLT-INSTRUMENTER: Total size of counters: 1260000 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 199375 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 10469868 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /home/boss/projects/oss/cpython/main/python.bolt
BOLT-INFO: 73572 instructions were shortened
BOLT-INFO: removed 83 empty blocks
BOLT-INFO: UCE removed 606 blocks and 36973 bytes of code
BOLT-INFO: padding code to 0x1200000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0x1401b80
BOLT-INFO: clear procedure is 0x13fd310
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting __bolt_runtime_start to 0x1401b80
BOLT-INFO: setting __bolt_runtime_fini to 0x1401bd0
BOLT-INFO: setting __hot_start to 0xa00000
BOLT-INFO: setting __hot_end to 0x11a3ff8
BOLT-INFO: runtime library finalization was hooked via DT_FINI, set to 0x1401bd0
BOLT-INFO: runtime library initialization was hooked via ELF Header Entry Point, set to 0x1401b80
# Run instrumented binaries to collect data.
./python -m test --pgo --timeout=
Using random seed: 707753353
0:00:00 load avg: 3.07 Run 43 tests sequentially in a single process
0:00:00 load avg: 3.07 [ 1/43] test_array
0:00:04 load avg: 2.91 [ 1/43] test_array passed
0:00:04 load avg: 2.91 [ 2/43] test_base64
0:00:05 load avg: 2.91 [ 2/43] test_base64 passed
0:00:05 load avg: 2.91 [ 3/43] test_binascii
0:00:06 load avg: 2.91 [ 3/43] test_binascii passed
0:00:06 load avg: 2.91 [ 4/43] test_binop
0:00:06 load avg: 2.91 [ 4/43] test_binop passed
0:00:06 load avg: 2.91 [ 5/43] test_bisect
0:00:06 load avg: 2.91 [ 5/43] test_bisect passed
0:00:06 load avg: 2.91 [ 6/43] test_bytes
0:00:16 load avg: 2.61 [ 6/43] test_bytes passed
0:00:16 load avg: 2.61 [ 7/43] test_bz2
0:00:17 load avg: 2.61 [ 7/43] test_bz2 passed
0:00:17 load avg: 2.61 [ 8/43] test_cmath
0:00:17 load avg: 2.61 [ 8/43] test_cmath passed
0:00:17 load avg: 2.61 [ 9/43] test_codecs
0:00:19 load avg: 2.48 [ 9/43] test_codecs passed
0:00:19 load avg: 2.48 [10/43] test_collections
0:00:21 load avg: 2.48 [10/43] test_collections passed
0:00:21 load avg: 2.48 [11/43] test_complex
0:00:21 load avg: 2.48 [11/43] test_complex passed
0:00:21 load avg: 2.48 [12/43] test_dataclasses
0:00:23 load avg: 2.36 [12/43] test_dataclasses passed
0:00:23 load avg: 2.36 [13/43] test_datetime
0:00:31 load avg: 2.25 [13/43] test_datetime passed
0:00:31 load avg: 2.25 [14/43] test_decimal
0:00:36 load avg: 2.15 [14/43] test_decimal passed
0:00:36 load avg: 2.15 [15/43] test_difflib
0:00:38 load avg: 2.15 [15/43] test_difflib passed
0:00:38 load avg: 2.15 [16/43] test_float
0:00:38 load avg: 2.06 [16/43] test_float passed
0:00:38 load avg: 2.06 [17/43] test_fstring
0:00:41 load avg: 2.06 [17/43] test_fstring passed
0:00:41 load avg: 2.06 [18/43] test_functools
Fatal Python error: Segmentation fault

Current thread 0x00007f3bb89cb780 [python] (most recent call first):
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib

Current thread's C stack trace (most recent call first):
  Binary file "./python", at _Py_DumpStack+0x3d [0x104acd1]
  Binary file "./python" [0x1045691]
  Binary file "./python" [0x10453ec]
  Binary file "/usr/lib/libc.so.6", at +0x3e6f0 [0x7f3bb863e6f0]
  Binary file "./python" [0x14032ab]

Extension modules: _testcapi, _testinternalcapi, _testlimitedcapi (total: 3)
make[1]: *** [Makefile:1054: profile-bolt-stamp] Segmentation fault (core dumped)
make[1]: Leaving directory '/home/boss/projects/oss/cpython/main'
make: *** [Makefile:1081: bolt-opt] Error 2

CPython versions tested on:

3.16, CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.16.0a0 (heads/main:42de93a62e7, Sep 7 2026, 13:34:10) [Clang 22.1.8 ]

[picnixz]

@Fidget-Spinner @corona10 @markshannon Is BOLT no longer experimental? could it be because we are exhausting the stack somehow (or maybe it's clang 22.1.8 that is the culprit...?)

@ashm-dev does it happen on 3.15?

[corona10]

I never consider that BOLT is non experimental, I need to reevaluate the current status.

[ashm-dev]

@picnixz Yes, the same issue occurs on both 3.15 and 3.14.

[Fidget-Spinner]

@ashm-dev pretty easy way to test if this is indeed a clang inlining too much and causing stack overflow: just pass --with-tail-call-interp to your configure. If now the tests pass, that points to a high stack consumption bug on normal interpreter builds.

[ashm-dev]

@Fidget-Spinner

CC=clang CXX=clang++ LDFLAGS='-fuse-ld=lld' ./configure --enable-bolt --enable-optimizations --with-tail-call-interp

BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: ca2b52862de907b4e4dca1d064098a28ab8269be
BOLT-INFO: first alloc address is 0x200000
BOLT-INFO: creating new program header table at address 0x800000, offset 0x600000
BOLT-INFO: enabling relocation mode
BOLT-INFO: forcing -jump-tables=move for instrumentation
BOLT-INFO: enabling lite mode
BOLT-INFO: 0 out of 9260 functions in the binary (0.0%) have non-empty execution profile
BOLT-INFO: validate-mem-refs updated 2 object references
BOLT-INSTRUMENTER: Number of indirect call site descriptors: 2305
BOLT-INSTRUMENTER: Number of indirect call target descriptors: 9216
BOLT-INSTRUMENTER: Number of function descriptors: 9216
BOLT-INSTRUMENTER: Number of branch counters: 106963
BOLT-INSTRUMENTER: Number of ST leaf node counters: 54155
BOLT-INSTRUMENTER: Number of direct call counters: 0
BOLT-INSTRUMENTER: Total number of counters: 161118
BOLT-INSTRUMENTER: Total size of counters: 1288944 bytes (static alloc memory)
BOLT-INSTRUMENTER: Total size of string table emitted: 205747 bytes in file
BOLT-INSTRUMENTER: Total size of descriptors: 10697116 bytes in file
BOLT-INSTRUMENTER: Profile will be saved to file /home/boss/projects/oss/cpython/main/python.bolt
BOLT-INFO: 75452 instructions were shortened
BOLT-INFO: removed 87 empty blocks
BOLT-INFO: UCE removed 615 blocks and 37565 bytes of code
BOLT-INFO: padding code to 0x1200000 to accommodate hot text
BOLT-INFO: output linked against instrumentation runtime library, lib entry point is 0x140db80
BOLT-INFO: clear procedure is 0x1409310
BOLT-INFO: patched build-id (flipped last bit)
BOLT-INFO: setting __bolt_runtime_start to 0x140db80
BOLT-INFO: setting __bolt_runtime_fini to 0x140dbd0
BOLT-INFO: setting __hot_start to 0xa00000
BOLT-INFO: setting __hot_end to 0x11d7378
BOLT-INFO: runtime library finalization was hooked via DT_FINI, set to 0x140dbd0
BOLT-INFO: runtime library initialization was hooked via ELF Header Entry Point, set to 0x140db80
# Run instrumented binaries to collect data.
./python -m test --pgo --timeout=
Using random seed: 2596992137
0:00:00 load avg: 2.84 Run 43 tests sequentially in a single process
0:00:00 load avg: 2.84 [ 1/43] test_array
0:00:04 load avg: 2.77 [ 1/43] test_array passed
0:00:04 load avg: 2.77 [ 2/43] test_base64
0:00:06 load avg: 2.77 [ 2/43] test_base64 passed
0:00:06 load avg: 2.77 [ 3/43] test_binascii
0:00:06 load avg: 2.77 [ 3/43] test_binascii passed
0:00:06 load avg: 2.77 [ 4/43] test_binop
0:00:07 load avg: 2.77 [ 4/43] test_binop passed
0:00:07 load avg: 2.77 [ 5/43] test_bisect
0:00:07 load avg: 2.77 [ 5/43] test_bisect passed
0:00:07 load avg: 2.77 [ 6/43] test_bytes
0:00:26 load avg: 2.48 [ 6/43] test_bytes passed
0:00:26 load avg: 2.48 [ 7/43] test_bz2
0:00:27 load avg: 2.48 [ 7/43] test_bz2 passed
0:00:27 load avg: 2.48 [ 8/43] test_cmath
0:00:27 load avg: 2.48 [ 8/43] test_cmath passed
0:00:27 load avg: 2.48 [ 9/43] test_codecs
0:00:30 load avg: 2.37 [ 9/43] test_codecs passed
0:00:30 load avg: 2.37 [10/43] test_collections
0:00:34 load avg: 2.37 [10/43] test_collections passed
0:00:34 load avg: 2.37 [11/43] test_complex
0:00:34 load avg: 2.26 [11/43] test_complex passed
0:00:34 load avg: 2.26 [12/43] test_dataclasses
0:00:36 load avg: 2.26 [12/43] test_dataclasses passed
0:00:36 load avg: 2.26 [13/43] test_datetime
0:00:49 load avg: 2.55 [13/43] test_datetime passed
0:00:49 load avg: 2.55 [14/43] test_decimal
0:00:57 load avg: 2.46 [14/43] test_decimal passed
0:00:57 load avg: 2.46 [15/43] test_difflib
0:01:00 load avg: 2.34 [15/43] test_difflib passed
0:01:00 load avg: 2.34 [16/43] test_float
0:01:00 load avg: 2.34 [16/43] test_float passed
0:01:00 load avg: 2.34 [17/43] test_fstring
0:01:04 load avg: 2.34 [17/43] test_fstring passed
0:01:04 load avg: 2.34 [18/43] test_functools
Fatal Python error: Segmentation fault

Current thread 0x00007f0e3e0a2780 [python] (most recent call first):
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib
  File "/home/boss/projects/oss/cpython/main/Lib/test/test_functools.py", line 2251 in fib

Current thread's C stack trace (most recent call first):
  Binary file "./python", at _Py_DumpStack+0x3d [0x1118dd1]
  Binary file "./python" [0x1113791]
  Binary file "./python" [0x11134ec]
  Binary file "/usr/lib/libc.so.6", at +0x3e6f0 [0x7f0e3de3e6f0]
  Binary file "./python" [0x140f2ab]

Extension modules: _testcapi, _testinternalcapi (total: 2)
make[1]: *** [Makefile:1054: profile-bolt-stamp] Segmentation fault (core dumped)
make[1]: Leaving directory '/home/boss/projects/oss/cpython/main'
make: *** [Makefile:1081: bolt-opt] Error 2