failed to self-sign certificate (Cannot allocate memory)

[webstean]

Starting baresip without a certificate with the dtls_srtp module enabled, generate an error
webstean@LSCPH-JLTNTC2:~/.baresip$ baresip
baresip v1.0.0 Copyright (C) 2010 - 2020 Alfred E. Heggestad et al.
Local network address: IPv4=eth0|172.26.140.145
aucodec: PCMU/8000/1
aucodec: PCMA/8000/1
aufilt: vumeter
auplay: pulse
ausrc: pulse
medianat: stun
medianat: turn
medianat: ice
mediaenc: srtp
mediaenc: srtp-mand
mediaenc: srtp-mandf
dtls_srtp: failed to self-sign certificate (Cannot allocate memory)
module dtls_srtp.so: Cannot allocate memory
Populated 1 account
Populated 8 contacts
Populated 2 audio codecs
Populated 1 audio filter
Populated 0 video codecs
Populated 0 video filters
baresip is ready.

[sreimers]

Can you provide further details? Like OS/Distro and especially the OpenSSL version?

[webstean]

--- System info: ---
Machine: x86_64/linux
Version: 1.0.0 (libre v0.6.1)
Build: 64-bit little endian
Kernel: Linux LSCPH-JLTNTC2 4.19.104-microsoft-standard #1 SMP Wed Feb 19 06:37:35 UTC 2020 x86_64
Uptime: 10 secs
Started: Tue Aug 11 17:08:50 2020
Compiler: 9.3.0
OpenSSL: OpenSSL 1.1.1f 31 Mar 2020

OS:/Distro: Ubunti under WSL. OpenSSL 1.1.1f

[alfredh]

the source of the error is:

tls: 140736434426816:error:140AB18F:SSL routines:SSL_CTX_use_certificate:ee key too small:ssl/ssl_rsa.c:310:

the RSA key generated is 1024, but your version of openssl requires 2048 bits or more:

see libre tls_set_selfsigned:

	if (!RSA_generate_key_ex(rsa, 1024, bn, NULL))
		goto out;

you can try to change the keysize to 2048 and see if that helps.

as a workaround you can also try to change the SECLEVEL in openssl.cnf:

https://serverfault.com/questions/957931/openssl-allow-usage-of-insecure-client-certs

[webstean]

Changed tls.c from 1024 to 2048 and that resolves the issue.

[webstean]

Created pull request
baresip/re#4 (comment)

[alfredh]

please note that increasing the RSA cert bits will impact the network traffic (DTLS)
as the Cert is sent as part of the DTLS handshake.

when you say that it resolves the issue, does that include testing call setup and DTLS handshake
with a remote endpoint ?

I dont think we should change the default RSA keysize, I think it is better to add a new function
that takes number of bits as input parameter. Example:

int tls_set_selfsigned_bits(struct tls *tls, const char *cn, size_t bits);

in the future we might want to increase it again, e.g. to 4096.

[oej]

In the future we want to move to elliptic curve certs which are smaller in size and lower in cpu usage.

[oej]

See https://thenewdialtone.com/webrtc-h-264-dtls/ - Chrome moved from 1024 bit RSA to ECDSA

"The previous algorithm (1024-bit RSA) takes about 1 second to process. Call setup time is an issue in VoIP in general due to ICE process and this is just another delay added. The new algorithm used (ECDSA) is fast, how fast? By the thousands faster"

So the answer is not 2048 bit RSA :-)

[alfredh]

I agree that we should also support ECDSA.

but we should also keep support for RSA, with a configurable bitsize.
we have to test that this works with the most used browsers.

[alfredh]

fixed in commit a9ceb87

please verify