Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with 8k keys and RSA while testing MariaDB #2904

Closed
dgarske opened this issue Apr 14, 2020 · 7 comments
Closed

Issue with 8k keys and RSA while testing MariaDB #2904

dgarske opened this issue Apr 14, 2020 · 7 comments
Assignees

Comments

@dgarske
Copy link
Contributor

dgarske commented Apr 14, 2020

Reported by @grooverdan:
Replaces PR #2901.

Discovered while resolving https://jira.mariadb.org/browse/MDEV-22221 / MariaDB/server#1501

Seems to be 8k certificates on the client/server side that doesn't occur on smaller sizes.

the server/client uses data from: https://github.com/MariaDB/server/tree/10.5/mysql-test/std_data

server:

key: server8k-key.pem
cert_file= server8k-cert.pem
ca_file=cacert.pem

client.key cert are:
client-cert.pem
client-key.pem

client side is a gnutls based.

I tried reproducing this with

examples/server/server -c /home/dan/repos/mariadb-server-10.4/mysql-test/std_data/server8k-cert.pem -k /home/dan/repos/mariadb-server-10.4/mysql-test/std_data/server8k-key.pem -A /home/dan/repos/mariadb-server-10.4/mysql-test/std_data/cacert.pem -i -d -v 4

however I was unable to generate the same results:

(from https://github.com/MariaDB/server/blob/10.4/mysql-test/main/ssl_8k_key.test)

Appears to be RSA:

This is wolfssl from the v4.3.0-stable tag.

https://gist.github.com/grooverdan/cadb9499c2e3adc0bfbb0d22663155ff

failing that:

git clone  --single-branch --branch 10.4-MDEV-22221-wolfssl-tls13 https://github.com/grooverdan/mariadb-server.git mariadb-server
cd mariadb-server
git submodule update --init --recursive
change mysql-test/main/ssl_8k_key.test to have TLSv1.3
mkdir  ../build
cd ../build
cmake -DPLUGIN_{ARCHIVE,TOKUDB,MROONGA,OQGRAPH,ROCKSDB,CONNECT,SPIDER}=NO   -DBUILD_CONFIG=mysql_release -DWITH_SSL=bundled  ../mariadb-server
make -j 8
mysql-test/mtr --mem {--manual-gdb} main.ssl_8k_key

Stack Trace:

stack_bottom = 0x7f3c6551ddd8 thread_stack 0x49000
/home/dan/repos/build-mariadb-server-10.4/sql/mysqld(my_print_stacktrace+0x29)[0x55e8ccd55de9]
mysys/stacktrace.c:270(my_print_stacktrace)[0x55e8cc7d617d]
sigaction.c:0(__restore_rt)[0x7f3c6bddfc70]
:0(__GI_raise)[0x7f3c6b6bfe35]
:0(__GI_abort)[0x7f3c6b6aa895]
:0(__libc_message)[0x7f3c6b70308f]
malloc.c:0(malloc_printerr)[0x7f3c6b70a40c]
malloc.c:0(munmap_chunk)[0x7f3c6b70a6bc]
/home/dan/repos/build-mariadb-server-10.4/sql/mysqld(+0xdab497)[0x55e8ccce1497]
src/tls13.c:5456(FreeScv13Args)[0x55e8ccce5876]
src/tls13.c:8605(wolfSSL_accept_TLSv13)[0x55e8ccda8d69]
vio/viossl.c:264(ssl_handshake_loop)[0x55e8cc563acc]
sql/sql_acl.cc:13020(parse_client_handshake_packet(MPVIO_EXT*, unsigned char**, unsigned long))[0x55e8cc5642be]
sql/sql_acl.cc:13365(server_mpvio_read_packet(st_plugin_vio*, unsigned char**))[0x55e8cc553735]
sql/sql_acl.cc:14048(native_password_authenticate(st_plugin_vio*, st_mysql_server_auth_info*))[0x55e8cc55539a]
sql/sql_acl.cc:13529(do_auth_once(THD*, st_mysql_const_lex_string const*, MPVIO_EXT*))[0x55e8cc568132]
sql/sql_acl.cc:13630(acl_authenticate(THD*, unsigned int))[0x55e8cc6bf650]
sql/sql_connect.cc:1080(check_connection(THD*))[0x55e8cc6bf8a2]
sql/sql_connect.cc:1150(login_connection(THD*))[0x55e8cc6c013a]
sql/sql_connect.cc:1324(thd_prepare_connection(THD*))[0x55e8cc6c05cf]
sql/sql_connect.cc:1404(do_handle_one_connection(CONNECT*))[0x55e8cc6c0732]
@dgarske
Copy link
Contributor Author

dgarske commented Apr 14, 2020

@grooverdan, can you make sure the following build option #define FP_MAX_BITS 16384 is defined? This will support 8K RSA keys.

@grooverdan
Copy link

yes. user_settings.h file:
https://gist.github.com/grooverdan/d4aab44bc1a99466a1790ae2c45e8b85
Is what's generated. Suggestions welcome.

Occurs both clang-8.0.0/gcc-9.3.1. I verified the same app->sigData memory address that is allocated is the same address that being freed. Also tested against wolfSSL master branch today with same error. Adding a stop point on munmap_chunk, _int_free, after SendTls13CertificateVerify was reached, showed a free in RsaUnPad_PSS however at a different address.

Better stacktrace (against wolfssl master branch)

Thread 7 "mysqld" received signal SIGABRT, Aborted.
0x00007ffff7861e35 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff7861e35 in raise () from /lib64/libc.so.6
#1  0x00007ffff784c895 in abort () from /lib64/libc.so.6
#2  0x00007ffff78a508f in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff78ac40c in malloc_printerr () from /lib64/libc.so.6
#4  0x00007ffff78add6c in _int_free () from /lib64/libc.so.6
#5  0x0000000000d680bb in FreeScv13Args () at /home/dan/repos/mariadb-server-10.4/extra/wolfssl/wolfssl/src/tls13.c:5455
#6  SendTls13CertificateVerify () at /home/dan/repos/mariadb-server-10.4/extra/wolfssl/wolfssl/src/tls13.c:5782
#7  0x0000000000d69a1c in wolfSSL_accept_TLSv13 () at /home/dan/repos/mariadb-server-10.4/extra/wolfssl/wolfssl/src/tls13.c:8605
#8  0x0000000000e448d6 in ssl_handshake_loop (vio=0x20c02a8, ssl=0x7fffd400fd30, func=0xd183b0 ) at /home/dan/repos/mariadb-server-10.4/vio/viossl.c:264
#9  ssl_do (ptr=, vio=0x20c02a8, timeout=, func=0xd183b0 , errptr=0x7ffff16bea58)
    at /home/dan/repos/mariadb-server-10.4/vio/viossl.c:315
#10 0x000000000064f8ae in parse_client_handshake_packet (mpvio=, buff=, pkt_len=)
    at /home/dan/repos/mariadb-server-10.4/sql/sql_acl.cc:13019
#11 server_mpvio_read_packet (param=0x7ffff16bf218, buf=0x7ffff16beed0) at /home/dan/repos/mariadb-server-10.4/sql/sql_acl.cc:13365
#12 0x0000000000653ed0 in native_password_authenticate (vio=0x7ffff16bf218, info=0x7ffff16bf230) at /home/dan/repos/mariadb-server-10.4/sql/sql_acl.cc:14048
#13 0x00000000006501f8 in do_auth_once (thd=0x7fffd4000c48, auth_plugin_name=, mpvio=0x7ffff16bf218) at /home/dan/repos/mariadb-server-10.4/sql/sql_acl.cc:13529
#14 0x000000000064dc72 in acl_authenticate (thd=0x7fffd4000c48, com_change_user_pkt_len=0) at /home/dan/repos/mariadb-server-10.4/sql/sql_acl.cc:13672
#15 0x000000000079ca1a in check_connection (thd=0x7fffd4000c48) at /home/dan/repos/mariadb-server-10.4/sql/sql_connect.cc:1080
#16 login_connection (thd=0x7fffd4000c48) at /home/dan/repos/mariadb-server-10.4/sql/sql_connect.cc:1149
#17 0x000000000079d534 in thd_prepare_connection (thd=0x7fffd4000c48) at /home/dan/repos/mariadb-server-10.4/sql/sql_connect.cc:1324
#18 0x000000000079d412 in do_handle_one_connection (connect=) at /home/dan/repos/mariadb-server-10.4/sql/sql_connect.cc:1402
#19 0x000000000079d263 in handle_one_connection (arg=0x216e428) at /home/dan/repos/mariadb-server-10.4/sql/sql_connect.cc:1316
#20 0x0000000000cf6d61 in pfs_spawn_thread (arg=0x20c04a8) at /home/dan/repos/mariadb-server-10.4/storage/perfschema/pfs.cc:1869
#21 0x00007ffff7f774c0 in start_thread () from /lib64/libpthread.so.0
#22 0x00007ffff7926133 in clone () from /lib64/libc.so.6

@dgarske
Copy link
Contributor Author

dgarske commented Apr 20, 2020

Hi @grooverdan,

The problem is one of the following:

  1. Misconfiguration between how wolfSSL is built vs. the application. Since you are using a user_settings.h, your wolfSSL library also needs to be built with it (or equivalent settings). Does your application define WOLFSSL_USER_SETTINGS? How are you building wolfSSL? If you are building wolfSSL with ./configure then you'll need ./configure CFLAGS="-DFP_MAX_BITS= 16384 and many other options. Also ./configure generates a build settings file wolfssl/options.h, which you could include in your application code above other wolf headers.
    We have a good FAQ for this here: https://www.wolfssl.com/docs/frequently-asked-questions-faq/#How_do_I_manage_the_build_configuration_for_wolfSSL?

  2. Insufficient stack space.

Thanks,
David Garske, wolfSSL

@grooverdan
Copy link

grooverdan commented Apr 27, 2020

Thanks to #2924 I can produce this with just wolfSSL:

 ./configure --enable-tls13 --enable-curve25519 --enable-hkdf --enable-maxstrength CFLAGS="-DFP_MAX_BITS=16384 -g"
make
LD_LIBRARY_PATH=$PWD/./src/.libs/  gdb --args examples/server/.libs/server -c /home/dan/repos/mariadb-server-10.4/mysql-test/std_data/server8k-cert.pem -k /home/dan/repos/mariadb-server-10.4/mysql-test/std_data/server8k-key.pem -A /home/dan/repos/mariadb-server-10.4/mysql-test/std_data/cacert.pem -i -d -v 4

Start client gnutls:

gnutls-cli --no-ca-verification --x509keyfile=client-certkey.pem --x509certfile=client-cert.pem -p 11111 localhost -d 5

or openssl:

openssl s_client -tls1_3 -connect localhost:11111 -cert client-cert.pem -key client-certkey.pem -debug

Resulting server backtrace:

munmap_chunk(): invalid pointer

Program received signal SIGABRT, Aborted.
0x00007ffff7c27e35 in raise () from /lib64/libc.so.6
(gdb) bt full
#0  0x00007ffff7c27e35 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007ffff7c12895 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x00007ffff7c6b08f in __libc_message () from /lib64/libc.so.6
No symbol table info available.
#3  0x00007ffff7c7240c in malloc_printerr () from /lib64/libc.so.6
No symbol table info available.
#4  0x00007ffff7c726bc in munmap_chunk () from /lib64/libc.so.6
No symbol table info available.
#5  0x00007ffff7facff7 in FreeScv13Args (ssl=0x40a6b0, pArgs=0x7fffffffd760) at src/tls13.c:5556
        xp = 
        args = 0x7fffffffd760
        args = 
        xp = 
#6  SendTls13CertificateVerify (ssl=ssl@entry=0x40a6b0) at src/tls13.c:5910
        ret = -132
        sig = 0x40a858
        args = {{output = 0x40b6b0 "\026\003\003\004\b\017", verify = 0x40b6b9 "\b\004\004", idx = 9, 
            sigLen = 1024, sendSz = 1037, length = 1024, sigAlgo = 8 '\b', 
            sigData = 0x40b990 "Z\311\020=+$\030v\354\305\350\307\b}0k\276\060vo7v\321\367\304*2\357W\252\201\353\242U\235-\356", , sigDataSz = 130}}
#7  0x00007ffff7fb0036 in wolfSSL_accept_TLSv13 (ssl=ssl@entry=0x40a6b0) at src/tls13.c:8788
No locals.
#8  0x00007ffff7fa3bb6 in wolfSSL_accept (ssl=ssl@entry=0x40a6b0) at src/ssl.c:11971
No locals.
#9  0x0000000000403cb3 in server_test (args=args@entry=0x7fffffffdae0) at examples/server/server.c:2236
        sockfd = 3
        clientfd = 4
        method = 
        ctx = 0x40a4b0
        ssl = 0x40a6b0
        msg = "I hear you fa shizzle!"
        useWebServerMsg = 
        input = "\240\v\333\367\377\177\000\000\220\002", '\000' , "M\204\307\367\377\177\000\000\000\000\000\000\000\000\000\000RJ\307\367\377\177", '\000' , "\002\000\000\000\000\000\000\000\016\000\000\000\000\000\000\200"
        ch = 
        version = 
        doCliCertCheck = 
        useAnyAddr = 
        port = 11111
        usePsk = 
--Type  for more, q to quit, c to continue without paging--
        usePskPlus = 
        doDTLS = 
        dtlsUDP = 
        useNtruKey = 
        nonBlocking = 
        fewerPackets = 
        wc_shutdown = 
        resume = 
        resumeCount = 
        loops = 
        cnt = 
        echoData = 
        block = 
        throughput = 
        minDhKeyBits = 
        doListen = 0
        ret = 
        err = 0
        serverReadyFile = 
        cipherList = 
        useDefCipherList = 
        overrideDateErrors = 
        verifyCert = 
        ourCert = 
        ourKey = 
        ourDhParam = 
        readySignal = 
        argc = 
        argv = 
        buffer = '\000' , "'\000\000\000\000\000\000\000)\000\000\000\000\000\000\000\200\002\000\000\000\000\000\000\240\v\333\367\377\177\000"
        noPskDheKe = 
        updateKeysIVs = 
        mutualAuth = 
        doDhKeyCheck = 
        onlyKeyShare = 
        useX25519 = 
        exitWithRet = 
        loadCertKeyIntoSSLObj = 
        disallowETM = 
        client = 
        client_len = 
#10 0x0000000000402586 in main (argc=, argv=) at examples/server/server.c:2555
--Type  for more, q to quit, c to continue without paging--
        args = {argc = 11, argv = 0x7fffffffdc68, return_code = -1, signal = 0x7fffffffdb10, callbacks = 0x0}
        ready = {ready = 0, port = 0, srfName = 0x0, mutex = {__data = {__lock = 0, __count = 0, __owner = 0, 
              __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
            __size = '\000' , __align = 0}, cond = {__data = {{__wseq = 0, __wseq32 = {__low = 0, 
                  __high = 0}}, {__g1_start = 0, __g1_start32 = {__low = 0, __high = 0}}, __g_refs = {0, 0}, 
              __g_size = {0, 0}, __g1_orig_size = 0, __wrefs = 0, __g_signals = {0, 0}}, 
            __size = '\000' , __align = 0}}

I haven't looked at the stack size yet.

@SparkiDev
Copy link
Contributor

Hi @grooverdan,

I believe I have a fix for the issue: #2933
I was able to reproduce the error and it went away after the fix.

Please test the patch and let us know.

Thanks,
Sean Parkinson, wolfSSL

@grooverdan
Copy link

fixed. Thanks.

@dgarske
Copy link
Contributor Author

dgarske commented May 4, 2020

Thanks @grooverdan and @SparkiDev . Marking this closed.

@dgarske dgarske closed this as completed May 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants