while enable https in http transport, MHD_handle_connection will crash

[jerrylining]

Version:
gnutls-3.5.19
nettle 3.4
Backtrace:

#0  0x00007fefb6727277 in raise () at /usr/lib64/libc.so.6
#1  0x00007fefb6728968 in abort () at /usr/lib64/libc.so.6
#2  0x00007fefb2556980 in __gmp_default_reallocate () at /usr/local/lib/libgmp.so.10
#3  0x00007fefb27cea82 in _nettle_gmp_alloc (n=n@entry=140664473911424) at gmp-glue.c:314
#4  0x00007fefb27c93bc in nettle_pkcs1_rsa_digest_encode (m=m@entry=0x7fef7a18d550, key_size=140664473911424, di_length=83, digest_info=digest_info@entry=0x7fef6c002ed0 "0Q0\r\006\t`\206H\001e\003\004\002\003\005")
    at pkcs1-rsa-digest.c:48
#5  0x00007fefb27ca439 in nettle_rsa_pkcs1_sign_tr (pub=pub@entry=0x7fef7a18d5d0, key=key@entry=0x7fef7a18d600, random_ctx=random_ctx@entry=0x0, random=random@entry=0x7fef889326a0 <rnd_func>, length=<optimized out>, digest_info=0x7fef6c002ed0 "0Q0\r\006\t`\206H\001e\003\004\002\003\005", s=s@entry=0x7fef7a18d5c0) at rsa-pkcs1-sign-tr.c:54
#6  0x00007fef88934271 in _wrap_nettle_pk_sign (algo=<optimized out>, signature=0x7fef7a18d910, vdata=0x7fef7a18d6f0, pk_params=0xf98070) at pk.c:426
#7  0x00007fef888b2d6a in gnutls_privkey_sign_hash (signer=0xf77580, hash_algo=
    GNUTLS_DIG_SHA512, flags=<optimized out>, hash_data=<optimized out>, signature=0x7fef7a18d910)
    at gnutls_privkey.c:794
#8  0x00007fef888a8ce3 in sign_tls_hash (session=session@entry=0x7fef68008aa0, hash_algo=hash_algo@entry=GNUTLS_DIG_SHA512, cert=cert@entry=0xf951c0, pkey=pkey@entry=0xf77580, hash_concat=hash_concat@entry=0x7fef7a18d7c0, signature=signature@entry=0x7fef7a18d910) at gnutls_sig.c:231
#9  0x00007fef888a9144 in _gnutls_handshake_sign_data (session=session@entry=0x7fef68008aa0, cert=0xf951c0, pkey=0xf77580, params=params@entry=0x7fef7a18d920, signature=signature@entry=0x7fef7a18d910, sign_algo=sign_algo@entry=0x7fef7a18d8fc) at gnutls_sig.c:154
#10 0x00007fef8890b05d in gen_dhe_server_kx (session=0x7fef68008aa0, data=0x7fef7a18d970) at dhe.c:191
#11 0x00007fef88899774 in _gnutls_send_server_kx_message (session=session@entry=0x7fef68008aa0, again=0)
    at gnutls_kx.c:180
#12 0x00007fef88898020 in gnutls_handshake (session=0x7fef68008aa0) at gnutls_handshake.c:3000
#13 0x00007fef88898020 in gnutls_handshake (session=0x7fef68008aa0) at gnutls_handshake.c:2455
#14 0x00007fef88b89c5c in run_tls_handshake () at /usr/lib64/libmicrohttpd.so.10
#15 0x00007fef88b89ce9 in MHD_tls_connection_handle_read () at /usr/lib64/libmicrohttpd.so.10
#16 0x00007fef88b81419 in MHD_handle_connection () at /usr/lib64/libmicrohttpd.so.10
#17 0x00007fefb7febe25 in start_thread () at /usr/lib64/libpthread.so.0
#18 0x00007fefb67efbad in clone () at /usr/lib64/libc.so.6

it seems caused by wrong private key size, I think there generate wrong key size value

[lminiero]

@jerrylining please STOP pasting logs as they are INLINE. The hashtag numbers are seen as references to past issues, and they create wrong links all over the place. I put (again) your snippet in a quote: please don't do it again or I'll be forced to immediately close your issues.

[lminiero]

I don't think it's a private key issue, or otherwise the setup of the HTTPS web server would fail at startup. Look more like an issue when handling an incoming HTTPS request. What libmicrohttpd version are you using? If older, have you tried upgrading it?

[jerrylining]

Previously I think it might be caused by wrong libnettle or libgnutls version, and then I have tried different versions of them. It occurred again. I will check libmicrohttpd version soon.

[jerrylining]

I switched microhttplibd to version libmicrohttpd-0.9.32, now the issue is gone. It is true that the older microhttplibd caused the issue. Issue fixed, Thank you :)

[jerrylining]

Issue Fixed in libmicrohttpd-0.9.32+