Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ThreadSanitizer reports lock-order-inversion in evp_extra_test #9454

Closed
mspncp opened this issue Jul 24, 2019 · 12 comments
Closed

ThreadSanitizer reports lock-order-inversion in evp_extra_test #9454

mspncp opened this issue Jul 24, 2019 · 12 comments
Labels
issue: bug report The issue was opened to report a bug

Comments

@mspncp
Copy link
Contributor

mspncp commented Jul 24, 2019 •
edited
Loading

@bernd-edlinger pointed me to his observation in #9437 (comment) that the ThreadSanitizer reports errors for the evp_extra_test. This here is only the first one reported.

It can be reproduced on current master (d0cf719) as follows:

./config -g -fsanitize=thread
make -j 16
TSAN_OPTIONS=second_deadlock_stack=1 OPENSSL_TEST_RAND_ORDER=1563972976 util/shlib_wrap.sh /home/msp/src/openssl/test/evp_extra_test

The first issue reported by the ThreadSanitizer is a lock-order-inversion caused by

openssl_ctx_get_data(), which takes the context lock

openssl/crypto/context.c

Line 195 in d0cf719

CRYPTO_THREAD_read_lock(ctx->lock);

and ossl_provider_forall_loaded() which takes the store lock

openssl/crypto/provider_core.c

Line 587 in d0cf719

CRYPTO_THREAD_read_lock(store->lock);

Although the issue occurs during the test, I think it reveals a fundamental locking problem IMHO.

Here are the call stacks. (Forget about the #frame numbers, but the source code locations are real links!)

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=32344)
Cycle in lock order graph: M44 (0x7b10000003c0) => M7 (0x7b1000000000) => M44

Mutex M7 acquired here while holding mutex M44 in main thread:
#0 pthread_rwlock_rdlock /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/tsan/tsan_interceptors.cc:1242 (libtsan.so.0+0x2d7e9)
#1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x206495)
#2 openssl_ctx_get_data crypto/context.c:195 (libcrypto.so.3+0x1f7059)
#3 ossl_namemap_stored crypto/core_namemap.c:92 (libcrypto.so.3+0x1f7c31)
#4 put_method_in_store crypto/evp/evp_fetch.c:126 (libcrypto.so.3+0x1d58af)
#5 ossl_method_construct_this crypto/core_fetch.c:55 (libcrypto.so.3+0x1f780a)
#6 algorithm_do_this crypto/core_algorithm.c:50 (libcrypto.so.3+0x1f748c)
#7 provider_forall_loaded crypto/provider_core.c:567 (libcrypto.so.3+0x2054ab)
#8 ossl_provider_forall_loaded crypto/provider_core.c:627 (libcrypto.so.3+0x2054ab)
#9 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f7679)
#10 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f7a6b)
#11 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d5d39)
#12 EVP_MD_fetch crypto/evp/digest.c:686 (libcrypto.so.3+0x1bad13)
#13 EVP_DigestInit_ex crypto/evp/digest.c:177 (libcrypto.so.3+0x1bb400)
#14 EVP_DigestInit crypto/evp/digest.c:106 (libcrypto.so.3+0x1bb49a)
#15 sm2_compute_z_digest crypto/sm2/sm2_sign.c:64 (libcrypto.so.3+0x287cd0)
#16 pkey_sm2_digest_custom crypto/sm2/sm2_pmeth.c:287 (libcrypto.so.3+0x286e34)
#17 do_sigver_init crypto/evp/m_sigver.c:83 (libcrypto.so.3+0x1dc979)
#18 EVP_DigestVerifyInit crypto/evp/m_sigver.c:97 (libcrypto.so.3+0x1dc979)
#19 test_EVP_SM2_verify test/evp_extra_test.c:664 (evp_extra_test+0x599b)
#20 run_tests test/testutil/driver.c:353 (evp_extra_test+0x8f0d)
#21 main test/testutil/main.c:30 (evp_extra_test+0x4c4a)

Mutex M44 previously acquired by the same thread here:
#0 pthread_rwlock_rdlock /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/tsan/tsan_interceptors.cc:1242 (libtsan.so.0+0x2d7e9)
#1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x206495)
#2 ossl_provider_forall_loaded crypto/provider_core.c:587 (libcrypto.so.3+0x2052e2)
#3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f7679)
#4 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f7a6b)
#5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d5d39)
#6 EVP_MD_fetch crypto/evp/digest.c:686 (libcrypto.so.3+0x1bad13)
#7 EVP_DigestInit_ex crypto/evp/digest.c:177 (libcrypto.so.3+0x1bb400)
#8 EVP_DigestInit crypto/evp/digest.c:106 (libcrypto.so.3+0x1bb49a)
#9 sm2_compute_z_digest crypto/sm2/sm2_sign.c:64 (libcrypto.so.3+0x287cd0)
#10 pkey_sm2_digest_custom crypto/sm2/sm2_pmeth.c:287 (libcrypto.so.3+0x286e34)
#11 do_sigver_init crypto/evp/m_sigver.c:83 (libcrypto.so.3+0x1dc979)
#12 EVP_DigestVerifyInit crypto/evp/m_sigver.c:97 (libcrypto.so.3+0x1dc979)
#13 test_EVP_SM2_verify test/evp_extra_test.c:664 (evp_extra_test+0x599b)
#14 run_tests test/testutil/driver.c:353 (evp_extra_test+0x8f0d)
#15 main test/testutil/main.c:30 (evp_extra_test+0x4c4a)

Mutex M44 acquired here while holding mutex M7 in main thread:
#0 pthread_rwlock_rdlock /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/tsan/tsan_interceptors.cc:1242 (libtsan.so.0+0x2d7e9)
#1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x206495)
#2 ossl_provider_forall_loaded crypto/provider_core.c:587 (libcrypto.so.3+0x2052e2)
#3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f7679)
#4 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f7a6b)
#5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d5d39)
#6 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (libcrypto.so.3+0x1d4993)
#7 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (libcrypto.so.3+0x242720)
#8 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (libcrypto.so.3+0x2445b4)
#9 rand_drbg_new crypto/rand/drbg_lib.c:442 (libcrypto.so.3+0x246242)
#10 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (libcrypto.so.3+0x246242)
#11 drbg_setup crypto/rand/drbg_lib.c:1120 (libcrypto.so.3+0x24672d)
#12 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (libcrypto.so.3+0x24672d)
#13 openssl_ctx_generic_new crypto/context.c:151 (libcrypto.so.3+0x1f6afe)
#14 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (libcrypto.so.3+0x1f9aeb)
#15 openssl_ctx_get_data crypto/context.c:204 (libcrypto.so.3+0x1f7082)
#16 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x246d4a)
#17 OPENSSL_CTX_get0_private_drbg crypto/rand/drbg_lib.c:1362 (libcrypto.so.3+0x246d4a)
#18 rand_priv_bytes_ex crypto/rand/rand_lib.c:814 (libcrypto.so.3+0x248b41)
#19 rand_priv_bytes_ex crypto/rand/rand_lib.c:805 (libcrypto.so.3+0x248b41)
#20 bnrand crypto/bn/bn_rand.c:51 (libcrypto.so.3+0xf0ff4)
#21 bnrand_range crypto/bn/bn_rand.c:182 (libcrypto.so.3+0xf0ff4)
#22 bnrand_range crypto/bn/bn_rand.c:212 (libcrypto.so.3+0xf1ff2)
#23 BN_priv_rand_range_ex crypto/bn/bn_rand.c:211 (libcrypto.so.3+0xf1ff2)
#24 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:324 (libcrypto.so.3+0xef871)
#25 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:272 (libcrypto.so.3+0xef871)
#26 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:246 (libcrypto.so.3+0xefdf4)
#27 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:203 (libcrypto.so.3+0xefdf4)
#28 BN_is_prime_ex crypto/bn/bn_prime.c:199 (libcrypto.so.3+0xefe90)
#29 RSA_check_key_ex crypto/rsa/rsa_chk.c:82 (libcrypto.so.3+0x252365)
#30 RSA_check_key_ex crypto/rsa/rsa_chk.c:25 (libcrypto.so.3+0x252365)
#31 rsa_pkey_check crypto/rsa/rsa_ameth.c:1032 (libcrypto.so.3+0x24ece1)
#32 EVP_PKEY_check crypto/evp/pmeth_gn.c:192 (libcrypto.so.3+0x1e3fe2)
#33 test_EVP_PKEY_check test/evp_extra_test.c:965 (evp_extra_test+0x532a)
#34 run_tests test/testutil/driver.c:386 (evp_extra_test+0x8d63)
#35 main test/testutil/main.c:30 (evp_extra_test+0x4c4a)

Mutex M7 previously acquired by the same thread here:
#0 pthread_rwlock_rdlock /var/tmp/portage/sys-devel/gcc-8.3.0-r1/work/gcc-8.3.0/libsanitizer/tsan/tsan_interceptors.cc:1242 (libtsan.so.0+0x2d7e9)
#1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x206495)
#2 openssl_ctx_get_data crypto/context.c:195 (libcrypto.so.3+0x1f7059)
#3 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x246d4a)
#4 OPENSSL_CTX_get0_private_drbg crypto/rand/drbg_lib.c:1362 (libcrypto.so.3+0x246d4a)
#5 rand_priv_bytes_ex crypto/rand/rand_lib.c:814 (libcrypto.so.3+0x248b41)
#6 rand_priv_bytes_ex crypto/rand/rand_lib.c:805 (libcrypto.so.3+0x248b41)
#7 bnrand crypto/bn/bn_rand.c:51 (libcrypto.so.3+0xf0ff4)
#8 bnrand_range crypto/bn/bn_rand.c:182 (libcrypto.so.3+0xf0ff4)
#9 bnrand_range crypto/bn/bn_rand.c:212 (libcrypto.so.3+0xf1ff2)
#10 BN_priv_rand_range_ex crypto/bn/bn_rand.c:211 (libcrypto.so.3+0xf1ff2)
#11 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:324 (libcrypto.so.3+0xef871)
#12 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:272 (libcrypto.so.3+0xef871)
#13 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:246 (libcrypto.so.3+0xefdf4)
#14 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:203 (libcrypto.so.3+0xefdf4)
#15 BN_is_prime_ex crypto/bn/bn_prime.c:199 (libcrypto.so.3+0xefe90)
#16 RSA_check_key_ex crypto/rsa/rsa_chk.c:82 (libcrypto.so.3+0x252365)
#17 RSA_check_key_ex crypto/rsa/rsa_chk.c:25 (libcrypto.so.3+0x252365)
#18 rsa_pkey_check crypto/rsa/rsa_ameth.c:1032 (libcrypto.so.3+0x24ece1)
#19 EVP_PKEY_check crypto/evp/pmeth_gn.c:192 (libcrypto.so.3+0x1e3fe2)
#20 test_EVP_PKEY_check test/evp_extra_test.c:965 (evp_extra_test+0x532a)
#21 run_tests test/testutil/driver.c:386 (evp_extra_test+0x8d63)
#22 main test/testutil/main.c:30 (evp_extra_test+0x4c4a)
@mspncp mspncp added the issue: bug report The issue was opened to report a bug label Jul 24, 2019
@mspncp mspncp mentioned this issue Jul 24, 2019
Closed
@mspncp
Copy link
Contributor Author

mspncp commented Jul 24, 2019

Side note for the curious: the handy links were produced from the original tsan output using the following query-replace-regexp command in Emacs:

(query-replace-regexp "\\([a-z0-9/_.]+\\):\\([0-9]+\\)" "[\\1:\\2](https://github.com/openssl/openssl/blob/master/\\1#L\\2)" nil nil nil nil nil)

@paulidale
Copy link
Contributor

Any chance of a quick check to see if #9477 fixes this too?
This looks suspiciously similar to the RAND initialisation loop of #9455

@bernd-edlinger
Copy link
Member

The lock order inversions still prevail, but the too deep recursive read locks
and the memory use after free seem to be fixed.

$ TSAN_OPTIONS=second_deadlock_stack=1 OPENSSL_TEST_RAND_ORDER=1563972976 make test V=1 TESTS=test_evp_extra
make depend && make _tests
make[1]: Entering directory '/home/ed/OPC/openssl'
make[1]: Leaving directory '/home/ed/OPC/openssl'
make[1]: Entering directory '/home/ed/OPC/openssl'
( cd test; \
  mkdir -p test-runs; \
  SRCTOP=../. \
  BLDTOP=../. \
  RESULT_D=test-runs \
  PERL="/usr/bin/perl" \
  EXE_EXT= \
  OPENSSL_ENGINES=`cd .././engines 2>/dev/null && pwd` \
  OPENSSL_DEBUG_MEMORY=on \
    /usr/bin/perl .././test/run_tests.pl test_evp_extra )
../test/recipes/30-test_evp_extra.t .. 
1..1
    # RAND SEED 1563972976
    # Subtest: /home/ed/OPC/openssl/test/evp_extra_test
    1..13
    ok 1 - test_EVP_DigestVerifyInit
    ok 2 - test_EVP_SM2_verify
        # Subtest: test_d2i_AutoPrivateKey
        1..3
        ok 1 - iteration 2
        ok 2 - iteration 1
        ok 3 - iteration 3
    ok 3 - test_d2i_AutoPrivateKey
    ok 4 - test_HKDF
    ok 5 - test_EVP_PKCS82PKEY
        # Subtest: test_EVP_MD_fetch
        1..5
        ok 1 - iteration 2
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=539)
  Cycle in lock order graph: M248 (0x7b1000000e80) => M242 (0x7b1000000380) => M248

  Mutex M242 acquired here while holding mutex M248 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 ossl_namemap_stored crypto/core_namemap.c:92 (fips.so+0x45612)
    #4 put_method_in_store crypto/evp/evp_fetch.c:126 (fips.so+0x410a0)
    #5 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #6 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #7 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #8 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #9 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #10 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #11 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #12 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #13 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #14 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #15 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #16 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #17 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #18 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #19 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #20 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M248 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 ossl_provider_forall_loaded crypto/provider_core.c:621 (fips.so+0x4c485)
    #3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #4 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #6 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #7 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #8 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #9 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #10 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #11 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #12 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #13 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #14 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M248 acquired here while holding mutex M242 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 ossl_provider_forall_loaded crypto/provider_core.c:621 (fips.so+0x4c485)
    #3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #4 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #6 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (fips.so+0x40e65)
    #7 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (fips.so+0x5aeeb)
    #8 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (fips.so+0x5cb34)
    #9 rand_drbg_new crypto/rand/drbg_lib.c:442 (fips.so+0x5e76e)
    #10 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (fips.so+0x5e76e)
    #11 drbg_setup crypto/rand/drbg_lib.c:1120 (fips.so+0x5ec86)
    #12 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (fips.so+0x5ec86)
    #13 openssl_ctx_generic_new crypto/context.c:151 (fips.so+0x447de)
    #14 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (fips.so+0x472db)
    #15 openssl_ctx_get_data crypto/context.c:204 (fips.so+0x44aa2)
    #16 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #17 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #18 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #19 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #20 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #21 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #22 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #23 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #24 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #25 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M242 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #4 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #5 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #6 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #7 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #8 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #9 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #10 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #11 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #12 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) crypto/threads_pthread.c:60 in CRYPTO_THREAD_read_lock
==================
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=539)
  Cycle in lock order graph: M245 (0x7b1000000ec0) => M242 (0x7b1000000380) => M245

  Mutex M242 acquired here while holding mutex M245 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 ossl_prop_defn_get crypto/property/defn_cache.c:74 (fips.so+0x54ebb)
    #4 ossl_method_store_add crypto/property/property.c:204 (fips.so+0x5579d)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (fips.so+0x41137)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #13 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #14 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #15 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #16 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #17 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #18 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #19 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #20 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #21 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M245 previously acquired by the same thread here:
    #0 pthread_rwlock_wrlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1324 (libtsan.so.0+0x319da)
    #1 CRYPTO_THREAD_write_lock crypto/threads_pthread.c:73 (fips.so+0x4d69a)
    #2 ossl_property_write_lock crypto/property/property.c:78 (fips.so+0x5541d)
    #3 ossl_property_write_lock crypto/property/property.c:76 (fips.so+0x5541d)
    #4 ossl_method_store_add crypto/property/property.c:202 (fips.so+0x556cd)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (fips.so+0x41137)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #13 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #14 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #15 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #16 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #17 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #18 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #19 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #20 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #21 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M245 acquired here while holding mutex M242 in main thread:
    #0 pthread_rwlock_wrlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1324 (libtsan.so.0+0x319da)
    #1 CRYPTO_THREAD_write_lock crypto/threads_pthread.c:73 (fips.so+0x4d69a)
    #2 ossl_property_write_lock crypto/property/property.c:78 (fips.so+0x5541d)
    #3 ossl_property_write_lock crypto/property/property.c:76 (fips.so+0x5541d)
    #4 ossl_method_store_add crypto/property/property.c:202 (fips.so+0x556cd)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (fips.so+0x41137)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #13 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (fips.so+0x40e65)
    #14 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (fips.so+0x5aeeb)
    #15 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (fips.so+0x5cb34)
    #16 rand_drbg_new crypto/rand/drbg_lib.c:442 (fips.so+0x5e76e)
    #17 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (fips.so+0x5e76e)
    #18 drbg_setup crypto/rand/drbg_lib.c:1120 (fips.so+0x5ec86)
    #19 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (fips.so+0x5ec86)
    #20 openssl_ctx_generic_new crypto/context.c:151 (fips.so+0x447de)
    #21 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (fips.so+0x472db)
    #22 openssl_ctx_get_data crypto/context.c:204 (fips.so+0x44aa2)
    #23 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #24 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #25 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #26 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #27 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #28 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #29 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #30 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #31 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #32 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M242 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #4 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #5 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #6 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #7 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #8 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #9 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #10 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #11 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #12 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) crypto/threads_pthread.c:60 in CRYPTO_THREAD_read_lock
==================
        ok 2 - iteration 4
        ok 3 - iteration 1
        ok 4 - iteration 3
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=539)
  Cycle in lock order graph: M589 (0x7b1000002240) => M583 (0x7b10000022c0) => M589

  Mutex M583 acquired here while holding mutex M589 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 ossl_namemap_stored crypto/core_namemap.c:92 (fips.so+0x45612)
    #4 put_method_in_store crypto/evp/evp_fetch.c:126 (fips.so+0x410a0)
    #5 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #6 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #7 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #8 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #9 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #10 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #11 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #12 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #13 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #14 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #15 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #16 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #17 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #18 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #19 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #20 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M589 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 ossl_provider_forall_loaded crypto/provider_core.c:621 (fips.so+0x4c485)
    #3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #4 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #6 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #7 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #8 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #9 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #10 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #11 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #12 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #13 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #14 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M589 acquired here while holding mutex M583 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 ossl_provider_forall_loaded crypto/provider_core.c:621 (fips.so+0x4c485)
    #3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #4 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #6 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (fips.so+0x40e65)
    #7 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (fips.so+0x5aeeb)
    #8 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (fips.so+0x5cb34)
    #9 rand_drbg_new crypto/rand/drbg_lib.c:442 (fips.so+0x5e76e)
    #10 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (fips.so+0x5e76e)
    #11 drbg_setup crypto/rand/drbg_lib.c:1120 (fips.so+0x5ec86)
    #12 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (fips.so+0x5ec86)
    #13 openssl_ctx_generic_new crypto/context.c:151 (fips.so+0x447de)
    #14 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (fips.so+0x472db)
    #15 openssl_ctx_get_data crypto/context.c:204 (fips.so+0x44aa2)
    #16 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #17 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #18 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #19 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #20 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #21 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #22 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #23 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #24 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #25 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M583 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #4 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #5 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #6 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #7 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #8 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #9 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #10 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #11 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #12 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) crypto/threads_pthread.c:60 in CRYPTO_THREAD_read_lock
==================
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=539)
  Cycle in lock order graph: M586 (0x7b1000002280) => M583 (0x7b10000022c0) => M586

  Mutex M583 acquired here while holding mutex M586 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 ossl_prop_defn_get crypto/property/defn_cache.c:74 (fips.so+0x54ebb)
    #4 ossl_method_store_add crypto/property/property.c:204 (fips.so+0x5579d)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (fips.so+0x41137)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #13 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #14 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #15 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #16 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #17 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #18 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #19 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #20 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #21 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M586 previously acquired by the same thread here:
    #0 pthread_rwlock_wrlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1324 (libtsan.so.0+0x319da)
    #1 CRYPTO_THREAD_write_lock crypto/threads_pthread.c:73 (fips.so+0x4d69a)
    #2 ossl_property_write_lock crypto/property/property.c:78 (fips.so+0x5541d)
    #3 ossl_property_write_lock crypto/property/property.c:76 (fips.so+0x5541d)
    #4 ossl_method_store_add crypto/property/property.c:202 (fips.so+0x556cd)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (fips.so+0x41137)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #13 EVP_MD_fetch crypto/evp/digest.c:686 (fips.so+0x3daa5)
    #14 dummy_evp_call providers/fips/fipsprov.c:91 (fips.so+0x88cfa)
    #15 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88cfa)
    #16 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #17 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #18 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #19 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #20 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #21 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M586 acquired here while holding mutex M583 in main thread:
    #0 pthread_rwlock_wrlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1324 (libtsan.so.0+0x319da)
    #1 CRYPTO_THREAD_write_lock crypto/threads_pthread.c:73 (fips.so+0x4d69a)
    #2 ossl_property_write_lock crypto/property/property.c:78 (fips.so+0x5541d)
    #3 ossl_property_write_lock crypto/property/property.c:76 (fips.so+0x5541d)
    #4 ossl_method_store_add crypto/property/property.c:202 (fips.so+0x556cd)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (fips.so+0x41137)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (fips.so+0x451e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (fips.so+0x44ea3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (fips.so+0x4c4f6)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (fips.so+0x4c4f6)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (fips.so+0x4506e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (fips.so+0x4542a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (fips.so+0x414e2)
    #13 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (fips.so+0x40e65)
    #14 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (fips.so+0x5aeeb)
    #15 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (fips.so+0x5cb34)
    #16 rand_drbg_new crypto/rand/drbg_lib.c:442 (fips.so+0x5e76e)
    #17 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (fips.so+0x5e76e)
    #18 drbg_setup crypto/rand/drbg_lib.c:1120 (fips.so+0x5ec86)
    #19 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (fips.so+0x5ec86)
    #20 openssl_ctx_generic_new crypto/context.c:151 (fips.so+0x447de)
    #21 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (fips.so+0x472db)
    #22 openssl_ctx_get_data crypto/context.c:204 (fips.so+0x44aa2)
    #23 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #24 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #25 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #26 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #27 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #28 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #29 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #30 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #31 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #32 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M583 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (fips.so+0x4d66a)
    #2 openssl_ctx_get_data crypto/context.c:195 (fips.so+0x44a79)
    #3 drbg_get_global crypto/rand/drbg_lib.c:245 (fips.so+0x5f04a)
    #4 OPENSSL_CTX_get0_public_drbg crypto/rand/drbg_lib.c:1334 (fips.so+0x5f04a)
    #5 dummy_evp_call providers/fips/fipsprov.c:104 (fips.so+0x88d6a)
    #6 OSSL_provider_init providers/fips/fipsprov.c:369 (fips.so+0x88d6a)
    #7 provider_activate crypto/provider_core.c:439 (libcrypto.so.3+0x205ec6)
    #8 ossl_provider_activate crypto/provider_core.c:532 (libcrypto.so.3+0x206da8)
    #9 OSSL_PROVIDER_load crypto/provider.c:24 (libcrypto.so.3+0x2055bf)
    #10 test_EVP_MD_fetch test/evp_extra_test.c:1128 (evp_extra_test+0x40752f)
    #11 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #12 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) crypto/threads_pthread.c:60 in CRYPTO_THREAD_read_lock
==================
        ok 5 - iteration 5
    ok 6 - test_EVP_MD_fetch
        # Subtest: test_EVP_PKEY_check
        1..5
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=539)
  Cycle in lock order graph: M47 (0x7b10000003c0) => M10 (0x7b1000000000) => M47

  Mutex M10 acquired here while holding mutex M47 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x20814a)
    #2 openssl_ctx_get_data crypto/context.c:195 (libcrypto.so.3+0x1f8d79)
    #3 ossl_namemap_stored crypto/core_namemap.c:92 (libcrypto.so.3+0x1f9912)
    #4 put_method_in_store crypto/evp/evp_fetch.c:126 (libcrypto.so.3+0x1d7140)
    #5 ossl_method_construct_this crypto/core_fetch.c:55 (libcrypto.so.3+0x1f94e6)
    #6 algorithm_do_this crypto/core_algorithm.c:50 (libcrypto.so.3+0x1f91a3)
    #7 provider_forall_loaded crypto/provider_core.c:567 (libcrypto.so.3+0x206f26)
    #8 ossl_provider_forall_loaded crypto/provider_core.c:628 (libcrypto.so.3+0x206f26)
    #9 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f936e)
    #10 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f972a)
    #11 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d7582)
    #12 EVP_MD_fetch crypto/evp/digest.c:686 (libcrypto.so.3+0x1bdb24)
    #13 EVP_DigestInit_ex crypto/evp/digest.c:177 (libcrypto.so.3+0x1be21f)
    #14 EVP_DigestInit crypto/evp/digest.c:106 (libcrypto.so.3+0x1be2bb)
    #15 sm2_compute_z_digest crypto/sm2/sm2_sign.c:64 (libcrypto.so.3+0x28847d)
    #16 pkey_sm2_digest_custom crypto/sm2/sm2_pmeth.c:287 (libcrypto.so.3+0x287651)
    #17 do_sigver_init crypto/evp/m_sigver.c:83 (libcrypto.so.3+0x1de29f)
    #18 EVP_DigestVerifyInit crypto/evp/m_sigver.c:97 (libcrypto.so.3+0x1de29f)
    #19 test_EVP_SM2_verify test/evp_extra_test.c:664 (evp_extra_test+0x405b5a)
    #20 run_tests test/testutil/driver.c:353 (evp_extra_test+0x408994)
    #21 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M47 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x20814a)
    #2 ossl_provider_forall_loaded crypto/provider_core.c:621 (libcrypto.so.3+0x206eb5)
    #3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f936e)
    #4 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f972a)
    #5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d7582)
    #6 EVP_MD_fetch crypto/evp/digest.c:686 (libcrypto.so.3+0x1bdb24)
    #7 EVP_DigestInit_ex crypto/evp/digest.c:177 (libcrypto.so.3+0x1be21f)
    #8 EVP_DigestInit crypto/evp/digest.c:106 (libcrypto.so.3+0x1be2bb)
    #9 sm2_compute_z_digest crypto/sm2/sm2_sign.c:64 (libcrypto.so.3+0x28847d)
    #10 pkey_sm2_digest_custom crypto/sm2/sm2_pmeth.c:287 (libcrypto.so.3+0x287651)
    #11 do_sigver_init crypto/evp/m_sigver.c:83 (libcrypto.so.3+0x1de29f)
    #12 EVP_DigestVerifyInit crypto/evp/m_sigver.c:97 (libcrypto.so.3+0x1de29f)
    #13 test_EVP_SM2_verify test/evp_extra_test.c:664 (evp_extra_test+0x405b5a)
    #14 run_tests test/testutil/driver.c:353 (evp_extra_test+0x408994)
    #15 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M47 acquired here while holding mutex M10 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x20814a)
    #2 ossl_provider_forall_loaded crypto/provider_core.c:621 (libcrypto.so.3+0x206eb5)
    #3 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f936e)
    #4 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f972a)
    #5 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d7582)
    #6 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (libcrypto.so.3+0x1d61f4)
    #7 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (libcrypto.so.3+0x2434db)
    #8 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (libcrypto.so.3+0x2451e4)
    #9 rand_drbg_new crypto/rand/drbg_lib.c:442 (libcrypto.so.3+0x246e1e)
    #10 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (libcrypto.so.3+0x246e1e)
    #11 drbg_setup crypto/rand/drbg_lib.c:1120 (libcrypto.so.3+0x247340)
    #12 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (libcrypto.so.3+0x247340)
    #13 openssl_ctx_generic_new crypto/context.c:151 (libcrypto.so.3+0x1f884e)
    #14 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (libcrypto.so.3+0x1fb77b)
    #15 openssl_ctx_get_data crypto/context.c:204 (libcrypto.so.3+0x1f8da2)
    #16 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x24799a)
    #17 OPENSSL_CTX_get0_private_drbg crypto/rand/drbg_lib.c:1362 (libcrypto.so.3+0x24799a)
    #18 rand_priv_bytes_ex crypto/rand/rand_lib.c:814 (libcrypto.so.3+0x249782)
    #19 rand_priv_bytes_ex crypto/rand/rand_lib.c:805 (libcrypto.so.3+0x249782)
    #20 bnrand crypto/bn/bn_rand.c:51 (libcrypto.so.3+0xf6304)
    #21 bnrand_range crypto/bn/bn_rand.c:182 (libcrypto.so.3+0xf6304)
    #22 bnrand_range crypto/bn/bn_rand.c:212 (libcrypto.so.3+0xf7372)
    #23 BN_priv_rand_range_ex crypto/bn/bn_rand.c:211 (libcrypto.so.3+0xf7372)
    #24 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:324 (libcrypto.so.3+0xf4b14)
    #25 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:272 (libcrypto.so.3+0xf4b14)
    #26 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:246 (libcrypto.so.3+0xf50b5)
    #27 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:203 (libcrypto.so.3+0xf50b5)
    #28 BN_is_prime_ex crypto/bn/bn_prime.c:199 (libcrypto.so.3+0xf5150)
    #29 RSA_check_key_ex crypto/rsa/rsa_chk.c:82 (libcrypto.so.3+0x252cf3)
    #30 RSA_check_key_ex crypto/rsa/rsa_chk.c:25 (libcrypto.so.3+0x252cf3)
    #31 rsa_pkey_check crypto/rsa/rsa_ameth.c:1032 (libcrypto.so.3+0x24f747)
    #32 EVP_PKEY_check crypto/evp/pmeth_gn.c:192 (libcrypto.so.3+0x1e58f3)
    #33 test_EVP_PKEY_check test/evp_extra_test.c:965 (evp_extra_test+0x4055e3)
    #34 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #35 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M10 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x20814a)
    #2 openssl_ctx_get_data crypto/context.c:195 (libcrypto.so.3+0x1f8d79)
    #3 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x24799a)
    #4 OPENSSL_CTX_get0_private_drbg crypto/rand/drbg_lib.c:1362 (libcrypto.so.3+0x24799a)
    #5 rand_priv_bytes_ex crypto/rand/rand_lib.c:814 (libcrypto.so.3+0x249782)
    #6 rand_priv_bytes_ex crypto/rand/rand_lib.c:805 (libcrypto.so.3+0x249782)
    #7 bnrand crypto/bn/bn_rand.c:51 (libcrypto.so.3+0xf6304)
    #8 bnrand_range crypto/bn/bn_rand.c:182 (libcrypto.so.3+0xf6304)
    #9 bnrand_range crypto/bn/bn_rand.c:212 (libcrypto.so.3+0xf7372)
    #10 BN_priv_rand_range_ex crypto/bn/bn_rand.c:211 (libcrypto.so.3+0xf7372)
    #11 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:324 (libcrypto.so.3+0xf4b14)
    #12 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:272 (libcrypto.so.3+0xf4b14)
    #13 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:246 (libcrypto.so.3+0xf50b5)
    #14 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:203 (libcrypto.so.3+0xf50b5)
    #15 BN_is_prime_ex crypto/bn/bn_prime.c:199 (libcrypto.so.3+0xf5150)
    #16 RSA_check_key_ex crypto/rsa/rsa_chk.c:82 (libcrypto.so.3+0x252cf3)
    #17 RSA_check_key_ex crypto/rsa/rsa_chk.c:25 (libcrypto.so.3+0x252cf3)
    #18 rsa_pkey_check crypto/rsa/rsa_ameth.c:1032 (libcrypto.so.3+0x24f747)
    #19 EVP_PKEY_check crypto/evp/pmeth_gn.c:192 (libcrypto.so.3+0x1e58f3)
    #20 test_EVP_PKEY_check test/evp_extra_test.c:965 (evp_extra_test+0x4055e3)
    #21 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #22 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) crypto/threads_pthread.c:60 in CRYPTO_THREAD_read_lock
==================
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=539)
  Cycle in lock order graph: M44 (0x7b1000000180) => M10 (0x7b1000000000) => M44

  Mutex M10 acquired here while holding mutex M44 in main thread:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x20814a)
    #2 openssl_ctx_get_data crypto/context.c:195 (libcrypto.so.3+0x1f8d79)
    #3 ossl_prop_defn_get crypto/property/defn_cache.c:74 (libcrypto.so.3+0x23d46b)
    #4 ossl_method_store_add crypto/property/property.c:204 (libcrypto.so.3+0x23dd4d)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (libcrypto.so.3+0x1d71d7)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (libcrypto.so.3+0x1f94e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (libcrypto.so.3+0x1f91a3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (libcrypto.so.3+0x206f26)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (libcrypto.so.3+0x206f26)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f936e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f972a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d7582)
    #13 EVP_MD_fetch crypto/evp/digest.c:686 (libcrypto.so.3+0x1bdb24)
    #14 EVP_DigestInit_ex crypto/evp/digest.c:177 (libcrypto.so.3+0x1be21f)
    #15 EVP_DigestInit crypto/evp/digest.c:106 (libcrypto.so.3+0x1be2bb)
    #16 sm2_compute_z_digest crypto/sm2/sm2_sign.c:64 (libcrypto.so.3+0x28847d)
    #17 pkey_sm2_digest_custom crypto/sm2/sm2_pmeth.c:287 (libcrypto.so.3+0x287651)
    #18 do_sigver_init crypto/evp/m_sigver.c:83 (libcrypto.so.3+0x1de29f)
    #19 EVP_DigestVerifyInit crypto/evp/m_sigver.c:97 (libcrypto.so.3+0x1de29f)
    #20 test_EVP_SM2_verify test/evp_extra_test.c:664 (evp_extra_test+0x405b5a)
    #21 run_tests test/testutil/driver.c:353 (evp_extra_test+0x408994)
    #22 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M44 previously acquired by the same thread here:
    #0 pthread_rwlock_wrlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1324 (libtsan.so.0+0x319da)
    #1 CRYPTO_THREAD_write_lock crypto/threads_pthread.c:73 (libcrypto.so.3+0x20817a)
    #2 ossl_property_write_lock crypto/property/property.c:78 (libcrypto.so.3+0x23d9cd)
    #3 ossl_property_write_lock crypto/property/property.c:76 (libcrypto.so.3+0x23d9cd)
    #4 ossl_method_store_add crypto/property/property.c:202 (libcrypto.so.3+0x23dc7d)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (libcrypto.so.3+0x1d71d7)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (libcrypto.so.3+0x1f94e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (libcrypto.so.3+0x1f91a3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (libcrypto.so.3+0x206f26)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (libcrypto.so.3+0x206f26)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f936e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f972a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d7582)
    #13 EVP_MD_fetch crypto/evp/digest.c:686 (libcrypto.so.3+0x1bdb24)
    #14 EVP_DigestInit_ex crypto/evp/digest.c:177 (libcrypto.so.3+0x1be21f)
    #15 EVP_DigestInit crypto/evp/digest.c:106 (libcrypto.so.3+0x1be2bb)
    #16 sm2_compute_z_digest crypto/sm2/sm2_sign.c:64 (libcrypto.so.3+0x28847d)
    #17 pkey_sm2_digest_custom crypto/sm2/sm2_pmeth.c:287 (libcrypto.so.3+0x287651)
    #18 do_sigver_init crypto/evp/m_sigver.c:83 (libcrypto.so.3+0x1de29f)
    #19 EVP_DigestVerifyInit crypto/evp/m_sigver.c:97 (libcrypto.so.3+0x1de29f)
    #20 test_EVP_SM2_verify test/evp_extra_test.c:664 (evp_extra_test+0x405b5a)
    #21 run_tests test/testutil/driver.c:353 (evp_extra_test+0x408994)
    #22 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M44 acquired here while holding mutex M10 in main thread:
    #0 pthread_rwlock_wrlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1324 (libtsan.so.0+0x319da)
    #1 CRYPTO_THREAD_write_lock crypto/threads_pthread.c:73 (libcrypto.so.3+0x20817a)
    #2 ossl_property_write_lock crypto/property/property.c:78 (libcrypto.so.3+0x23d9cd)
    #3 ossl_property_write_lock crypto/property/property.c:76 (libcrypto.so.3+0x23d9cd)
    #4 ossl_method_store_add crypto/property/property.c:202 (libcrypto.so.3+0x23dc7d)
    #5 put_method_in_store crypto/evp/evp_fetch.c:136 (libcrypto.so.3+0x1d71d7)
    #6 ossl_method_construct_this crypto/core_fetch.c:55 (libcrypto.so.3+0x1f94e6)
    #7 algorithm_do_this crypto/core_algorithm.c:50 (libcrypto.so.3+0x1f91a3)
    #8 provider_forall_loaded crypto/provider_core.c:567 (libcrypto.so.3+0x206f26)
    #9 ossl_provider_forall_loaded crypto/provider_core.c:628 (libcrypto.so.3+0x206f26)
    #10 ossl_algorithm_do_all crypto/core_algorithm.c:72 (libcrypto.so.3+0x1f936e)
    #11 ossl_method_construct crypto/core_fetch.c:92 (libcrypto.so.3+0x1f972a)
    #12 evp_generic_fetch crypto/evp/evp_fetch.c:210 (libcrypto.so.3+0x1d7582)
    #13 EVP_CIPHER_fetch crypto/evp/evp_enc.c:1248 (libcrypto.so.3+0x1d61f4)
    #14 drbg_ctr_init crypto/rand/drbg_ctr.c:389 (libcrypto.so.3+0x2434db)
    #15 RAND_DRBG_set crypto/rand/drbg_lib.c:333 (libcrypto.so.3+0x2451e4)
    #16 rand_drbg_new crypto/rand/drbg_lib.c:442 (libcrypto.so.3+0x246e1e)
    #17 RAND_DRBG_secure_new_ex crypto/rand/drbg_lib.c:481 (libcrypto.so.3+0x246e1e)
    #18 drbg_setup crypto/rand/drbg_lib.c:1120 (libcrypto.so.3+0x247340)
    #19 drbg_ossl_ctx_new crypto/rand/drbg_lib.c:175 (libcrypto.so.3+0x247340)
    #20 openssl_ctx_generic_new crypto/context.c:151 (libcrypto.so.3+0x1f884e)
    #21 CRYPTO_alloc_ex_data crypto/ex_data.c:424 (libcrypto.so.3+0x1fb77b)
    #22 openssl_ctx_get_data crypto/context.c:204 (libcrypto.so.3+0x1f8da2)
    #23 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x24799a)
    #24 OPENSSL_CTX_get0_private_drbg crypto/rand/drbg_lib.c:1362 (libcrypto.so.3+0x24799a)
    #25 rand_priv_bytes_ex crypto/rand/rand_lib.c:814 (libcrypto.so.3+0x249782)
    #26 rand_priv_bytes_ex crypto/rand/rand_lib.c:805 (libcrypto.so.3+0x249782)
    #27 bnrand crypto/bn/bn_rand.c:51 (libcrypto.so.3+0xf6304)
    #28 bnrand_range crypto/bn/bn_rand.c:182 (libcrypto.so.3+0xf6304)
    #29 bnrand_range crypto/bn/bn_rand.c:212 (libcrypto.so.3+0xf7372)
    #30 BN_priv_rand_range_ex crypto/bn/bn_rand.c:211 (libcrypto.so.3+0xf7372)
    #31 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:324 (libcrypto.so.3+0xf4b14)
    #32 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:272 (libcrypto.so.3+0xf4b14)
    #33 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:246 (libcrypto.so.3+0xf50b5)
    #34 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:203 (libcrypto.so.3+0xf50b5)
    #35 BN_is_prime_ex crypto/bn/bn_prime.c:199 (libcrypto.so.3+0xf5150)
    #36 RSA_check_key_ex crypto/rsa/rsa_chk.c:82 (libcrypto.so.3+0x252cf3)
    #37 RSA_check_key_ex crypto/rsa/rsa_chk.c:25 (libcrypto.so.3+0x252cf3)
    #38 rsa_pkey_check crypto/rsa/rsa_ameth.c:1032 (libcrypto.so.3+0x24f747)
    #39 EVP_PKEY_check crypto/evp/pmeth_gn.c:192 (libcrypto.so.3+0x1e58f3)
    #40 test_EVP_PKEY_check test/evp_extra_test.c:965 (evp_extra_test+0x4055e3)
    #41 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #42 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

  Mutex M10 previously acquired by the same thread here:
    #0 pthread_rwlock_rdlock ../../../../gcc-10-20190721/libsanitizer/tsan/tsan_interceptors.cc:1294 (libtsan.so.0+0x3171a)
    #1 CRYPTO_THREAD_read_lock crypto/threads_pthread.c:60 (libcrypto.so.3+0x20814a)
    #2 openssl_ctx_get_data crypto/context.c:195 (libcrypto.so.3+0x1f8d79)
    #3 drbg_get_global crypto/rand/drbg_lib.c:245 (libcrypto.so.3+0x24799a)
    #4 OPENSSL_CTX_get0_private_drbg crypto/rand/drbg_lib.c:1362 (libcrypto.so.3+0x24799a)
    #5 rand_priv_bytes_ex crypto/rand/rand_lib.c:814 (libcrypto.so.3+0x249782)
    #6 rand_priv_bytes_ex crypto/rand/rand_lib.c:805 (libcrypto.so.3+0x249782)
    #7 bnrand crypto/bn/bn_rand.c:51 (libcrypto.so.3+0xf6304)
    #8 bnrand_range crypto/bn/bn_rand.c:182 (libcrypto.so.3+0xf6304)
    #9 bnrand_range crypto/bn/bn_rand.c:212 (libcrypto.so.3+0xf7372)
    #10 BN_priv_rand_range_ex crypto/bn/bn_rand.c:211 (libcrypto.so.3+0xf7372)
    #11 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:324 (libcrypto.so.3+0xf4b14)
    #12 bn_miller_rabin_is_prime crypto/bn/bn_prime.c:272 (libcrypto.so.3+0xf4b14)
    #13 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:246 (libcrypto.so.3+0xf50b5)
    #14 BN_is_prime_fasttest_ex crypto/bn/bn_prime.c:203 (libcrypto.so.3+0xf50b5)
    #15 BN_is_prime_ex crypto/bn/bn_prime.c:199 (libcrypto.so.3+0xf5150)
    #16 RSA_check_key_ex crypto/rsa/rsa_chk.c:82 (libcrypto.so.3+0x252cf3)
    #17 RSA_check_key_ex crypto/rsa/rsa_chk.c:25 (libcrypto.so.3+0x252cf3)
    #18 rsa_pkey_check crypto/rsa/rsa_ameth.c:1032 (libcrypto.so.3+0x24f747)
    #19 EVP_PKEY_check crypto/evp/pmeth_gn.c:192 (libcrypto.so.3+0x1e58f3)
    #20 test_EVP_PKEY_check test/evp_extra_test.c:965 (evp_extra_test+0x4055e3)
    #21 run_tests test/testutil/driver.c:386 (evp_extra_test+0x4087bb)
    #22 main test/testutil/main.c:30 (evp_extra_test+0x404c38)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) crypto/threads_pthread.c:60 in CRYPTO_THREAD_read_lock
==================
        ok 1 - iteration 2
        ok 2 - iteration 4
        ok 3 - iteration 1
        ok 4 - iteration 3
        ok 5 - iteration 5
    ok 7 - test_EVP_PKEY_check
        # Subtest: test_invalide_ec_char2_pub_range_decode
        1..3
        ok 1 - iteration 2
        ok 2 - iteration 1
        ok 3 - iteration 3
    ok 8 - test_invalide_ec_char2_pub_range_decode
        # Subtest: test_set_get_raw_keys
        1..7
        ok 1 - iteration 3
        ok 2 - iteration 6
        ok 3 - iteration 2
        ok 4 - iteration 5
        ok 5 - iteration 1
        ok 6 - iteration 4
        ok 7 - iteration 7
    ok 9 - test_set_get_raw_keys
    ok 10 - test_EVP_DigestSignInit
    ok 11 - test_X509_PUBKEY_inplace
    ok 12 - test_EVP_Enveloped
    ok 13 - test_EVP_SM2
ThreadSanitizer: reported 6 warnings
/home/ed/OPC/openssl/util/shlib_wrap.sh /home/ed/OPC/openssl/test/evp_extra_test => 66
not ok 1 - running evp_extra_test

#   Failed test 'running evp_extra_test'
#   at ../test/recipes/30-test_evp_extra.t line 21.
# Looks like you failed 1 test of 1.
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests 

Test Summary Report
-------------------
../test/recipes/30-test_evp_extra.t (Wstat: 256 Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=1, Tests=1,  1 wallclock secs ( 0.03 usr  0.00 sys +  0.14 cusr  0.02 csys =  0.19 CPU)
Result: FAIL
Makefile:198: recipe for target '_tests' failed
make[1]: *** [_tests] Error 1
make[1]: Leaving directory '/home/ed/OPC/openssl'
Makefile:196: recipe for target 'tests' failed
make: *** [tests] Error 2

@bernd-edlinger
Copy link
Member

bernd-edlinger commented Jul 29, 2019 •
edited
Loading

My guess is that all providers need to document their dependencies before
they are loaded, and not just on demand.
The loader should make a graph out of it and sort it top down, and make sure
all dependencies are satisfied (and already initialized) before initializing each provider.

@paulidale
Copy link
Contributor

Thanks for the output. The test ran okay for me whereas previously it deadlocked hence my optimism. I'll try to have another look at this one tomorrow.

I'm not sure what to do about the dependency graph. This test case jumps all over the place with respect to algorithm usage (& hence initialisation). Having the graph and initialising in a topologically sorted order won't necessarily address the problem with the lock inversion. E.g. initialising a public key algorithm then symmetric ciphers could start one way whereas reversing these could be another. Always initialising everything would work but it would be a big change from the current lazy model and one I'm not sure how palatable it would be. A single global initialisation lock might be the solution.

@bernd-edlinger
Copy link
Member

bernd-edlinger commented Jul 29, 2019 •
edited
Loading

I have of course not a ready solution. I just have the impression that it is not at all
clear why a aes-ctr provider is not allowed to use a rand provider. Or when you add
later add a replacement aes-ctr provider that is not aware of the possible recursion,
how will we know that it is incompatible with the rand-provider, maybe also replacement,
using more ciphers than the original one.

I mean from application code, it is probably not possible to know in advance what it will need.
But I think from a rand or aes-ctr provider we should expect a manifest describing all possible
dependencies so it is clear from the beginning that we have something incompatible.

@paulidale
Copy link
Contributor

So long as the algorithm is fetched rather than being used, it should work. The CTR DRBG vs AES-CTR is a good example BTW.

@bernd-edlinger
Copy link
Member

The drbg and aes-ctr provider is good today, but not so clear what will be added in the future.
What is not okay, is recursive new_func callbacks, with mutex held.

That may work today, but not when we do not know what new_func is actually doing.
So what would happen if the provider is a third-party component?

Why is there a recursion in the new_func ?

diff --git a/crypto/context.c b/crypto/context.c
index cc1ec0e..1fb7074 100644
--- a/crypto/context.c
+++ b/crypto/context.c
@@ -148,7 +148,11 @@ static void openssl_ctx_generic_new(void *parent_ign, void *ptr_ign,
                                     long argl_ign, void *argp)
 {
     const OPENSSL_CTX_METHOD *meth = argp;
+static int recurs = 0;
+recurs++;
+OPENSSL_assert(recurs == 1);
     void *ptr = meth->new_func(crypto_ex_data_get_openssl_ctx(ad));
+recurs--;
 
     if (ptr != NULL)
         CRYPTO_set_ex_data(ad, index, ptr);

It would not need recursion, if the CTR is initialized before RAND
and RAND is initialized before RSA, right?

@paulidale
Copy link
Contributor

@levitte @mattcaswell thoughts on the recursion question?

@paulidale
Copy link
Contributor

Rather than initilisation, isn't it more that CTR DRBG needs an AES CTR cipher context?

When a call is made to get data from the DRBG, the caller won't know what other cryptographic algorithm will be required. It's up to the specific DRBG to determine this.

This is looking more problematic the more I dig in.

@paulidale
Copy link
Contributor

I've emailed the project list for wider discussion.

This was referenced Aug 1, 2019
Closed
Closed
mattcaswell added a commit to mattcaswell/openssl that referenced this issue Aug 14, 2019
@mattcaswell
Fix context locking
847cf49 
Some parts of OPENSSL_CTX intialisation can get quite complex (e.g. RAND).
This can lead to complex interactions where different parts of the library
try to initialise while other parts are still initialising. This can lead
to deadlocks because both parts want to obtain the init lock.

We separate out the init lock so that it is only used to manage the
dynamic list of indexes. Each part of the library gets its own
initialisation lock.

Fixes openssl#9454
@mattcaswell mattcaswell mentioned this issue Aug 14, 2019
Closed
@mxmauro
Copy link

mxmauro commented Aug 27, 2019

Today I hit the same deadlock in my app.

Thread 1:

RtlEnterCriticalSection
CRYPTO_THREAD_read_lock
openssl_ctx_get_data
ossl_namemap_stored
put_method_in_store
ossl_method_construct_this
algorithm_do_this
provider_forall_loaded
ossl_provider_forall_loaded
ossl_algorithm_do_all
ossl_method_construct
evp_generic_fetch
EVP_MD_fetch
EVP_DigestInit_ex
PKCS12_key_gen_uni
PKCS12_key_gen_utf8
pkcs12_gen_mac
PKCS12_verify_mac
PKCS12_parse

Thread 2:

RtlEnterCriticalSection
CRYPTO_THREAD_read_lock
ossl_provider_forall_loaded
ossl_algorithm_do_all
ossl_method_construct
evp_generic_fetch
EVP_CIPHER_fetch
drbg_ctr_init
RAND_DRBG_set
rand_drbg_new
RAND_DRBG_secure_new_ex
drbg_setup
drbg_ossl_ctx_new
openssl_ctx_generic_new
CRYPTO_alloc_ex_data
openssl_ctx_get_data
drbg_get_global
OPENSSL_CTX_get0_public_drbg
rand_bytes_ex
RAND_bytes
SSL_CTX_new

Kind regards,
Mauro.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
issue: bug report The issue was opened to report a bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove the context lock. paulidale/openssl
Fix context locking mattcaswell/openssl
4 participants
@mxmauro @mspncp @bernd-edlinger @paulidale