test_hashlib fails when run on Free Threading with TSAN

[vstinner]

Crash report

TSAN CI job: https://github.com/python/cpython/actions/runs/28747943876/job/85377269260?pr=152717

test_hashlib fails when run on Free Threading with TSAN on the main branch.

Commands:

export CC=clang
export TSAN_OPTIONS="$PWD/log_path=san_log suppressions=$PWD/Tools/tsan/suppressions_free_threading.txt handle_segv=0"

./configure --config-cache --with-thread-sanitizer --with-pydebug --disable-gil
make -j4
./python -m test --tsan-parallel --parallel-threads=4 -j4 -W --timeout=600 --slowest

Tests output (truncated):

Using random seed: 1463135961
0:00:00 load avg: 2.65 mem: 267.1 MiB Run 2 tests in parallel using 2 worker processes (timeout: 10 min, worker timeout: 15 min)
0:00:06 load avg: 2.76 mem: 611.2 MiB [1/2] test_abc passed
0:00:14 load avg: 2.78 mem: 278.3 MiB [2/2/1] test_hashlib worker non-zero exit code (Exit code 66)
test_algorithms_available (test.test_hashlib.HashLibTestCase.test_algorithms_available) [threads=4] ... ok
test_algorithms_guaranteed (test.test_hashlib.HashLibTestCase.test_algorithms_guaranteed) [threads=4] ... ok
(...)
test_scrypt_types (test.test_hashlib.TestScrypt.test_scrypt_types) [threads=4] ... ok
test_scrypt_validate (test.test_hashlib.TestScrypt.test_scrypt_validate) [threads=4] ... ok

----------------------------------------------------------------------
Ran 88 tests in 9.698s

OK (skipped=56)

== Tests result: FAILURE ==

TSAN logs (truncated):

WARNING: ThreadSanitizer: data race (pid=30628)
  Read of size 8 at 0x721c0000e3a8 by thread T456:
    #0 memcmp <null> (python+0x10d49e) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)
    #1 <null> <null> (libcrypto.so.3+0x24f2d1) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
    #2 pbkdf2_hmac /home/runner/work/cpython/cpython/./Modules/clinic/_hashopenssl.c.h:1490:20 (_hashlib.cpython-316td-x86_64-linux-gnu.so+0x7299) (BuildId: 1c2fdd533a623cb5e65ca60ec9662a58487ffb8b)

  Previous write of size 8 at 0x721c0000e3a8 by thread T454 (mutexes: write M0, write M1):
    #0 memcpy <null> (python+0xfc022) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)
    #1 <null> <null> (libcrypto.so.3+0x24f0e6) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
    #2 pbkdf2_hmac /home/runner/work/cpython/cpython/./Modules/clinic/_hashopenssl.c.h:1490:20 (_hashlib.cpython-316td-x86_64-linux-gnu.so+0x7299) (BuildId: 1c2fdd533a623cb5e65ca60ec9662a58487ffb8b)

  Mutex M0 (0x721000000500) created at:
    #0 pthread_rwlock_init <null> (python+0x103e6e) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)
    #1 CRYPTO_THREAD_lock_new <null> (libcrypto.so.3+0x275485) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
    #2 CRYPTO_THREAD_run_once <null> (libcrypto.so.3+0x27556c) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
    #3 run_exec_func /home/runner/work/cpython/cpython/Objects/moduleobject.c:707:15 (python+0x374a4a) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)

  Mutex M1 (0x7210000008c0) created at:
    #0 pthread_rwlock_init <null> (python+0x103e6e) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)
    #1 CRYPTO_THREAD_lock_new <null> (libcrypto.so.3+0x275485) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
    #2 CRYPTO_THREAD_run_once <null> (libcrypto.so.3+0x27556c) (BuildId: 6608be85412b580cc4873b0421a014a9cc2a7b41)
    #3 run_exec_func /home/runner/work/cpython/cpython/Objects/moduleobject.c:707:15 (python+0x374a4a) (BuildId: 5899a6ff881ea0db9e292325f4f3bbeaba9715e6)

Full TSAN logs: tsan_full.log.

[vstinner]

I can reproduce the issue with "OpenSSL 3.5.7 9 Jun 2026" on Fedora 44 with commands:

export TSAN_OPTIONS="$PWD/log_path=san_log suppressions=$PWD/Tools/tsan/suppressions_free_threading.txt handle_segv=0"
./python -u -m test --parallel-threads=4 -v test_hashlib -m KDFTests 2>&1|tee log

It's the test_pbkdf2_hmac_c() test which triggers the data race.

[vstinner]

See also issue gh-143750.

[vstinner]

I tried but failed (so far) to reproduce the issue with OpenSSL built with TSAN.

I applied this patch (extracted from #143750):

diff --git a/Tools/ssl/multissltests.py b/Tools/ssl/multissltests.py
index 1a213187b89..d5e38993d97 100755
--- a/Tools/ssl/multissltests.py
+++ b/Tools/ssl/multissltests.py
@@ -163,6 +163,12 @@
     dest='keep_sources',
     help="Keep original sources for debugging."
 )
+parser.add_argument(
+    '--tsan',
+    action='store_true',
+    dest='tsan',
+    help="Build with thread sanitizer. (Disables fips in OpenSSL 3.x)."
+)
 
 
 class AbstractBuilder(object):
@@ -317,6 +323,8 @@ def _build_src(self, config_args=()):
         """Now build openssl"""
         log.info("Running build in {}".format(self.build_dir))
         cwd = self.build_dir
+        if self.args.tsan:
+            config_args += ("-fsanitize=thread",)
         cmd = [
             "./config", *config_args,
             "shared", "--debug",

I ran commands:

./Tools/ssl/multissltests.py --steps=library --base-directory ~/multissl --openssl 3.5.7 --system Linux --tsan
export LD_LIBRARY_PATH=~/multissl/openssl/3.5.7/lib/ 

./configure --with-thread-sanitizer --with-pydebug --disable-gil --with-openssl=/home/vstinner/multissl/openssl/3.5.7
make clean
make

./python -u -m test --parallel-threads=4 -v test_hashlib -m KDFTests 2>&1|tee log

Sadly, so far I failed to reproduce the test_hashlib failure this way.