Add Reproducible Error Injection [1.1.1] #18356
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This is the version for the 1.1.1 branch. |
|
This is definitely a new feature (although just feature of the test framework) so not for 1.1.1. |
|
@bernd-edlinger If you want to have a 1.1.1 feature fork I'd suggest to create it as a separate GitHub project. |
This adds reproducible memory error and test-data error injection, to the fuzzy-test framework. This feature can be enabled with ./config -DERROR_INJECT and additionally to enable call stacks -DERROR_CALLSTACK If enable-asan is used, the callstack is printed by the sanitizer, otherwise please set a breakpoint at the function "break_here", which is executed each time a memory allocation error is injected. If called with the environment variable ERROR_INJECT defined to the empty string the initialization value is printed, and can be used to reproduce the failure later, by passing the value to the ERROR_INJECT variable. There is a search script that can be used to look for errors, and print the command to reproduce the bug: ./testrun.sh This runs in endless mode until an error is found.
bernd-edlinger
force-pushed
the
add_error_injection_1
branch
from
cb6c88c
to
ec286b2
Compare
bernd-edlinger
added a commit
to bernd-edlinger/openssl
that referenced
this pull request
Jul 5, 2022
This was found by my Reproducible Error Injection patch (openssl#18356) Due to the exact location of the injected memory error the sha256 digest is missing, and this causes much later the memory leak (and a failed assertion) in tls13_generate_secret. But the reproduction is a bit challenging, as it requires AESNI and RDRAND capability. OPENSSL_ia32cap=0x4200000000000000 ERROR_INJECT=1657070330 ../util/shlib_wrap.sh ./client-test ./corpora/client/791afc153e17db072175eeef85385a38d7f6d194 #0 0x7fceaffb7d4f in __sanitizer_print_stack_trace ../../../../src/libsanitizer/asan/asan_stack.cc:36 openssl#1 0x55fb9117f934 in my_malloc fuzz/test-corpus.c:114 openssl#2 0x7fceafa147f3 in OPENSSL_LH_insert crypto/lhash/lhash.c:109 openssl#3 0x7fceafa42639 in lh_OBJ_NAME_insert crypto/objects/obj_local.h:12 openssl#4 0x7fceafa42639 in OBJ_NAME_add crypto/objects/o_names.c:236 openssl#5 0x7fceaf9f7baa in EVP_add_digest crypto/evp/names.c:39 openssl#6 0x7fceaf9c6b97 in openssl_add_all_digests_int crypto/evp/c_alld.c:39 openssl#7 0x7fceafa0f8ec in ossl_init_add_all_digests crypto/init.c:275 openssl#8 0x7fceafa0f8ec in ossl_init_add_all_digests_ossl_ crypto/init.c:264 openssl#9 0x7fceaf69b4de in __pthread_once_slow /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_once.c:116 openssl#10 0x7fceafafb27c in CRYPTO_THREAD_run_once crypto/threads_pthread.c:118 openssl#11 0x7fceafa1000e in OPENSSL_init_crypto crypto/init.c:677 openssl#12 0x7fceafa1000e in OPENSSL_init_crypto crypto/init.c:611 openssl#13 0x7fceafdad3e8 in OPENSSL_init_ssl ssl/ssl_init.c:190 openssl#14 0x55fb9117ee0f in FuzzerInitialize fuzz/client.c:46 openssl#15 0x55fb9117e939 in main fuzz/test-corpus.c:194 openssl#16 0x7fceaf4bc082 in __libc_start_main ../csu/libc-start.c:308 openssl#17 0x55fb9117ec7d in _start (.../openssl/fuzz/client-test+0x2c7d) #0 0x7fceaffb7d4f in __sanitizer_print_stack_trace ../../../../src/libsanitizer/asan/asan_stack.cc:36 openssl#1 0x55fb9117f934 in my_malloc fuzz/test-corpus.c:114 openssl#2 0x7fceafa147f3 in OPENSSL_LH_insert crypto/lhash/lhash.c:109 openssl#3 0x7fceafa42639 in lh_OBJ_NAME_insert crypto/objects/obj_local.h:12 openssl#4 0x7fceafa42639 in OBJ_NAME_add crypto/objects/o_names.c:236 openssl#5 0x7fceaf9f7baa in EVP_add_digest crypto/evp/names.c:39 openssl#6 0x7fceafdad328 in ossl_init_ssl_base ssl/ssl_init.c:87 openssl#7 0x7fceafdad328 in ossl_init_ssl_base_ossl_ ssl/ssl_init.c:24 openssl#8 0x7fceaf69b4de in __pthread_once_slow /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_once.c:116 openssl#9 0x7fceafafb27c in CRYPTO_THREAD_run_once crypto/threads_pthread.c:118 openssl#10 0x7fceafdad412 in OPENSSL_init_ssl ssl/ssl_init.c:193 openssl#11 0x55fb9117ee0f in FuzzerInitialize fuzz/client.c:46 openssl#12 0x55fb9117e939 in main fuzz/test-corpus.c:194 openssl#13 0x7fceaf4bc082 in __libc_start_main ../csu/libc-start.c:308 openssl#14 0x55fb9117ec7d in _start (.../openssl/fuzz/client-test+0x2c7d) ================================================================= ==1320996==ERROR: LeakSanitizer: detected memory leaks Direct leak of 80 byte(s) in 1 object(s) allocated from: #0 0x7fceaffaa808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144 openssl#1 0x7fceafa19425 in CRYPTO_zalloc crypto/mem.c:230 openssl#2 0x7fceafa03a85 in int_ctx_new crypto/evp/pmeth_lib.c:144 openssl#3 0x7fceafa03a85 in EVP_PKEY_CTX_new_id crypto/evp/pmeth_lib.c:250 openssl#4 0x7fceafe38de5 in tls13_generate_secret ssl/tls13_enc.c:174 openssl#5 0x7fceafd9537f in ssl_derive ssl/s3_lib.c:4833 openssl#6 0x7fceafdde91c in tls_parse_stoc_key_share ssl/statem/extensions_clnt.c:1902 openssl#7 0x7fceafdd4ac1 in tls_parse_all_extensions ssl/statem/extensions.c:752 openssl#8 0x7fceafdf8079 in tls_process_server_hello ssl/statem/statem_clnt.c:1698 openssl#9 0x7fceafe01f87 in ossl_statem_client_process_message ssl/statem/statem_clnt.c:1034 openssl#10 0x7fceafdeec0d in read_state_machine ssl/statem/statem.c:636 openssl#11 0x7fceafdeec0d in state_machine ssl/statem/statem.c:434 openssl#12 0x7fceafdb88d7 in SSL_do_handshake ssl/ssl_lib.c:3718 openssl#13 0x55fb9117f07c in FuzzerTestOneInput fuzz/client.c:98 openssl#14 0x55fb9117f463 in testfile fuzz/test-corpus.c:182 openssl#15 0x55fb9117eb92 in main fuzz/test-corpus.c:226 openssl#16 0x7fceaf4bc082 in __libc_start_main ../csu/libc-start.c:308 Indirect leak of 1080 byte(s) in 1 object(s) allocated from: #0 0x7fceaffaa808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144 openssl#1 0x7fceafa19425 in CRYPTO_zalloc crypto/mem.c:230 openssl#2 0x7fceafa11555 in pkey_hkdf_init crypto/kdf/hkdf.c:51 openssl#3 0x7fceafa03b36 in int_ctx_new crypto/evp/pmeth_lib.c:160 openssl#4 0x7fceafa03b36 in EVP_PKEY_CTX_new_id crypto/evp/pmeth_lib.c:250 openssl#5 0x7fceafe38de5 in tls13_generate_secret ssl/tls13_enc.c:174 openssl#6 0x7fceafd9537f in ssl_derive ssl/s3_lib.c:4833 openssl#7 0x7fceafdde91c in tls_parse_stoc_key_share ssl/statem/extensions_clnt.c:1902 openssl#8 0x7fceafdd4ac1 in tls_parse_all_extensions ssl/statem/extensions.c:752 openssl#9 0x7fceafdf8079 in tls_process_server_hello ssl/statem/statem_clnt.c:1698 openssl#10 0x7fceafe01f87 in ossl_statem_client_process_message ssl/statem/statem_clnt.c:1034 openssl#11 0x7fceafdeec0d in read_state_machine ssl/statem/statem.c:636 openssl#12 0x7fceafdeec0d in state_machine ssl/statem/statem.c:434 openssl#13 0x7fceafdb88d7 in SSL_do_handshake ssl/ssl_lib.c:3718 openssl#14 0x55fb9117f07c in FuzzerTestOneInput fuzz/client.c:98 openssl#15 0x55fb9117f463 in testfile fuzz/test-corpus.c:182 openssl#16 0x55fb9117eb92 in main fuzz/test-corpus.c:226 openssl#17 0x7fceaf4bc082 in __libc_start_main ../csu/libc-start.c:308 SUMMARY: AddressSanitizer: 1160 byte(s) leaked in 2 allocation(s).
2 tasks
openssl-machine
pushed a commit
that referenced
this pull request
Jul 7, 2022
This was found by my Reproducible Error Injection patch (#18356) Due to the exact location of the injected memory error the sha256 digest is missing, and this causes much later the memory leak (and a failed assertion) in tls13_generate_secret. But the reproduction is a bit challenging, as it requires AESNI and RDRAND capability. OPENSSL_ia32cap=0x4200000000000000 ERROR_INJECT=1657070330 ../util/shlib_wrap.sh ./client-test ./corpora/client/791afc153e17db072175eeef85385a38d7f6d194 #0 0x7fceaffb7d4f in __sanitizer_print_stack_trace ../../../../src/libsanitizer/asan/asan_stack.cc:36 #1 0x55fb9117f934 in my_malloc fuzz/test-corpus.c:114 #2 0x7fceafa147f3 in OPENSSL_LH_insert crypto/lhash/lhash.c:109 #3 0x7fceafa42639 in lh_OBJ_NAME_insert crypto/objects/obj_local.h:12 #4 0x7fceafa42639 in OBJ_NAME_add crypto/objects/o_names.c:236 #5 0x7fceaf9f7baa in EVP_add_digest crypto/evp/names.c:39 #6 0x7fceaf9c6b97 in openssl_add_all_digests_int crypto/evp/c_alld.c:39 #7 0x7fceafa0f8ec in ossl_init_add_all_digests crypto/init.c:275 #8 0x7fceafa0f8ec in ossl_init_add_all_digests_ossl_ crypto/init.c:264 #9 0x7fceaf69b4de in __pthread_once_slow /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_once.c:116 #10 0x7fceafafb27c in CRYPTO_THREAD_run_once crypto/threads_pthread.c:118 #11 0x7fceafa1000e in OPENSSL_init_crypto crypto/init.c:677 #12 0x7fceafa1000e in OPENSSL_init_crypto crypto/init.c:611 #13 0x7fceafdad3e8 in OPENSSL_init_ssl ssl/ssl_init.c:190 #14 0x55fb9117ee0f in FuzzerInitialize fuzz/client.c:46 #15 0x55fb9117e939 in main fuzz/test-corpus.c:194 #16 0x7fceaf4bc082 in __libc_start_main ../csu/libc-start.c:308 #17 0x55fb9117ec7d in _start (.../openssl/fuzz/client-test+0x2c7d) #0 0x7fceaffb7d4f in __sanitizer_print_stack_trace ../../../../src/libsanitizer/asan/asan_stack.cc:36 #1 0x55fb9117f934 in my_malloc fuzz/test-corpus.c:114 #2 0x7fceafa147f3 in OPENSSL_LH_insert crypto/lhash/lhash.c:109 #3 0x7fceafa42639 in lh_OBJ_NAME_insert crypto/objects/obj_local.h:12 #4 0x7fceafa42639 in OBJ_NAME_add crypto/objects/o_names.c:236 #5 0x7fceaf9f7baa in EVP_add_digest crypto/evp/names.c:39 #6 0x7fceafdad328 in ossl_init_ssl_base ssl/ssl_init.c:87 #7 0x7fceafdad328 in ossl_init_ssl_base_ossl_ ssl/ssl_init.c:24 #8 0x7fceaf69b4de in __pthread_once_slow /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_once.c:116 #9 0x7fceafafb27c in CRYPTO_THREAD_run_once crypto/threads_pthread.c:118 #10 0x7fceafdad412 in OPENSSL_init_ssl ssl/ssl_init.c:193 #11 0x55fb9117ee0f in FuzzerInitialize fuzz/client.c:46 #12 0x55fb9117e939 in main fuzz/test-corpus.c:194 #13 0x7fceaf4bc082 in __libc_start_main ../csu/libc-start.c:308 #14 0x55fb9117ec7d in _start (.../openssl/fuzz/client-test+0x2c7d) ================================================================= ==1320996==ERROR: LeakSanitizer: detected memory leaks Direct leak of 80 byte(s) in 1 object(s) allocated from: #0 0x7fceaffaa808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144 #1 0x7fceafa19425 in CRYPTO_zalloc crypto/mem.c:230 #2 0x7fceafa03a85 in int_ctx_new crypto/evp/pmeth_lib.c:144 #3 0x7fceafa03a85 in EVP_PKEY_CTX_new_id crypto/evp/pmeth_lib.c:250 #4 0x7fceafe38de5 in tls13_generate_secret ssl/tls13_enc.c:174 #5 0x7fceafd9537f in ssl_derive ssl/s3_lib.c:4833 #6 0x7fceafdde91c in tls_parse_stoc_key_share ssl/statem/extensions_clnt.c:1902 #7 0x7fceafdd4ac1 in tls_parse_all_extensions ssl/statem/extensions.c:752 #8 0x7fceafdf8079 in tls_process_server_hello ssl/statem/statem_clnt.c:1698 #9 0x7fceafe01f87 in ossl_statem_client_process_message ssl/statem/statem_clnt.c:1034 #10 0x7fceafdeec0d in read_state_machine ssl/statem/statem.c:636 #11 0x7fceafdeec0d in state_machine ssl/statem/statem.c:434 #12 0x7fceafdb88d7 in SSL_do_handshake ssl/ssl_lib.c:3718 #13 0x55fb9117f07c in FuzzerTestOneInput fuzz/client.c:98 #14 0x55fb9117f463 in testfile fuzz/test-corpus.c:182 #15 0x55fb9117eb92 in main fuzz/test-corpus.c:226 #16 0x7fceaf4bc082 in __libc_start_main ../csu/libc-start.c:308 Indirect leak of 1080 byte(s) in 1 object(s) allocated from: #0 0x7fceaffaa808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144 #1 0x7fceafa19425 in CRYPTO_zalloc crypto/mem.c:230 #2 0x7fceafa11555 in pkey_hkdf_init crypto/kdf/hkdf.c:51 #3 0x7fceafa03b36 in int_ctx_new crypto/evp/pmeth_lib.c:160 #4 0x7fceafa03b36 in EVP_PKEY_CTX_new_id crypto/evp/pmeth_lib.c:250 #5 0x7fceafe38de5 in tls13_generate_secret ssl/tls13_enc.c:174 #6 0x7fceafd9537f in ssl_derive ssl/s3_lib.c:4833 #7 0x7fceafdde91c in tls_parse_stoc_key_share ssl/statem/extensions_clnt.c:1902 #8 0x7fceafdd4ac1 in tls_parse_all_extensions ssl/statem/extensions.c:752 #9 0x7fceafdf8079 in tls_process_server_hello ssl/statem/statem_clnt.c:1698 #10 0x7fceafe01f87 in ossl_statem_client_process_message ssl/statem/statem_clnt.c:1034 #11 0x7fceafdeec0d in read_state_machine ssl/statem/statem.c:636 #12 0x7fceafdeec0d in state_machine ssl/statem/statem.c:434 #13 0x7fceafdb88d7 in SSL_do_handshake ssl/ssl_lib.c:3718 #14 0x55fb9117f07c in FuzzerTestOneInput fuzz/client.c:98 #15 0x55fb9117f463 in testfile fuzz/test-corpus.c:182 #16 0x55fb9117eb92 in main fuzz/test-corpus.c:226 #17 0x7fceaf4bc082 in __libc_start_main ../csu/libc-start.c:308 SUMMARY: AddressSanitizer: 1160 byte(s) leaked in 2 allocation(s). Reviewed-by: Todd Short <todd.short@me.com> Reviewed-by: Shane Lontis <shane.lontis@oracle.com> Reviewed-by: Hugo Landau <hlandau@openssl.org> (Merged from #18725)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds reproducible memory error and test-data
error injection, to the fuzzy-test framework.
This feature can be enabled with ./config -DERROR_INJECT
and additionally to enable call stacks -DERROR_CALLSTACK
If enable-asan is used, the callstack is printed by the
sanitizer, otherwise please set a breakpoint
at the function "break_here", which is executed each
time a memory allocation error is injected.
If called with the environment variable ERROR_INJECT
defined to the empty string the initialization value
is printed, and can be used to reproduce the failure
later, by passing the value to the ERROR_INJECT variable.
There is a search script that can be used to look
for errors, and print the command to reproduce the bug:
./testrun.sh
This runs in endless mode until an error is found.
Checklist