TLS certificate decode error / ee key to small with tls_threads_mode = 1

[xadhoom]

Description

While trying latest kamailio 5.7 branch, when tls_threads_mode is set to 1, it fails to load self signed certificates. Setting tls_threads_mode to 0 works as expected. Certificates are self signed for a local test env, generated with openssl 3.x.

Troubleshooting

The issue is very similar to #3737 but in my case the openssl config seems correct, and happens only enabling the tls_threads_mode

Reproduction

Certs have been generated with openssl req -new -newkey rsa:4096 -x509 -sha256 -days 3650 -nodes -out server.pem -keyout server.key

server.pem.txt
server.key.txt

(these are self signed cert for testing, nothing that cannot be shared)

My tls.cfg is very simple:

[server:default]
method = TLSv1.2+
verify_certificate = no
require_certificate = no
private_key = /etc/kamailio/server.key
certificate = /etc/kamailio/server.pem


[client:default]
method = TLSv1.2+
verify_certificate = no
require_certificate = no

Log Messages

 1(35) NOTICE: tls [tls_domain.c:1168]: ksr_tls_fix_domain(): registered server_name callback handler for socket [:0], server_name='<default>' ...
 1(35) ERROR: tls [tls_domain.c:590]: load_cert(): TLSs<default>: Unable to load certificate file '/etc/kamailio/server.pem'
 1(35) ERROR: tls [tls_util.h:49]: tls_err_ret(): load_cert:error:03000072:digital envelope routines::decode error (sni: unknown)
 1(35) ERROR: tls [tls_util.h:49]: tls_err_ret(): load_cert:error:0A00018F:SSL routines::ee key too small (sni: unknown)
 1(35) ERROR: <core> [core/sr_module.c:913]: init_mod_child(): error while initializing module tls (/usr/lib/x86_64-linux-gnu/kamailio/modules/tls.so)

Possible Solutions

Don't use tls_threads_mode for now.

Additional Information

• Kamailio Version - output of kamailio -v

version: kamailio 5.7.4 (x86_64/linux) a0dfb8
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: a0dfb8 
compiled with gcc 11.4.0

Actually this is built from 5.7 branch, on commit a0dfb8c

• Operating System:

Containerized Ubunu jammy, updated as of today.

[xadhoom]

After further digging in my setup, as soon as I disabled the mqtt module everything works.

So there's maybe someting in the mqtt module that's not yet adapted to the new tls setup?

[space88man]

After further digging in my setup, as soon as I disabled the mqtt module everything works.

So there's maybe someting in the mqtt module that's not yet adapted to the new tls setup?

That's good info - can you put the mqtt config here - maybe it is using SSL.

[xadhoom]

Yes, mqtt is using TLS, the config is pretty trivial

modparam("mqtt", "host", MQTT_HOST)
modparam("mqtt", "port", 8883)
modparam("mqtt", "keepalive", 5)
modparam("mqtt", "id", MQTT_NODE_NAME)
modparam("mqtt", "username", MQTT_USER)
modparam("mqtt", "password", MQTT_PASS)
modparam("mqtt", "will_topic", "kamailio123")
modparam("mqtt", "will", "gone")
modparam("mqtt", "verify_certificate", 0)
// this will enable TLS
modparam("mqtt", "ca_file", "/ssl/ca.crt")

[space88man]

Does it make an immediate connection to the broker or does that only happen during operations - i.e., after startup but before handling traffic is there a connection to 8883?

[xadhoom]

it makes an immediate connection to the broker on startup

[space88man]

it makes an immediate connection to the broker on startup

Can you run kamailio under gdb with tls_threads_mode=1 with the following script
You will need to start kamailio first as thread 1 is not started yet something like

gdb --args /usr/local/sbin/kamailo ................
break main
r

then use the following script:

break CRYPTO_THREAD_set_local thread 1
commands
bt 16
cont
end

...then continue

[xadhoom]

I'll try, since my setup is dockerized I'll need to play a bit with it. Will report as soon as I'm able to do it, should not be too hard.

[space88man]

I can't reproduce any error with the config below.

#!define MQTT_HOST "test.mosquitto.org"
loadmodule "mqtt.so"
modparam("mqtt", "host", MQTT_HOST)
modparam("mqtt", "port", 8883)
modparam("mqtt", "keepalive", 5)
#modparam("mqtt", "id", MQTT_NODE_NAME)
#modparam("mqtt", "username", MQTT_USER)
#modparam("mqtt", "password", MQTT_PASS)
modparam("mqtt", "will_topic", "kamailio123")
modparam("mqtt", "will", "gone")
modparam("mqtt", "verify_certificate", 0)
// download crt file from https://test.mosquitto.org/ssl/mosquitto.org.crt
modparam("mqtt", "ca_file", "/etc/kamailio/certs/mosquitto.org.crt")

I do see exactly one TCP connection from a worker

.... 91.121.93.94:8883  users:(("kamailio",pid=86882,fd=14))

When I did a tcpdump on this connection it shows TLS - no errors in logs.

[xadhoom]

I've tried and gdb stops at

Thread 1 "kamailio" hit Breakpoint 2, 0x00007ff30a1c2a70 in CRYPTO_THREAD_set_local () from /lib/x86_64-linux-gnu/libcrypto.so.3
(gdb) bt

[xadhoom]

I can't reproduce any error with the config below.

well, I have various other modules loaded, but only disabling mqtt makes it work, that's why I pointed at it.

[space88man]

Can you print the backtrace when it stops? After you type bt it should print out the call stack

[xadhoom]

Yep, sorry

Thread 1 "kamailio" hit Breakpoint 2, 0x00007fb1301c2a70 in CRYPTO_THREAD_set_local () from /lib/x86_64-linux-gnu/libcrypto.so.3
#0  0x00007fb1301c2a70 in CRYPTO_THREAD_set_local () from /lib/x86_64-linux-gnu/libcrypto.so.3
#1  0x00007fb1301bfad3 in OPENSSL_thread_stop () from /lib/x86_64-linux-gnu/libcrypto.so.3
#2  0x00007fb1301bfb43 in OPENSSL_cleanup () from /lib/x86_64-linux-gnu/libcrypto.so.3
#3  0x00005638fca17d48 in destroy_tls () at core/tls_hooks.c:75
#4  cleanup (show_status=1) at /usr/local/src/pkg/src/main.c:595
#5  0x00005638fce45a8d in shutdown_children.constprop.0 (show_status=show_status@entry=1, sig=15) at /usr/local/src/pkg/src/main.c:722
#6  0x00005638fca12cd5 in handle_sigs () at /usr/local/src/pkg/src/main.c:822
#7  0x00005638fca1baa4 in main_loop () at /usr/local/src/pkg/src/main.c:1989
#8  0x00005638fca0cffc in main (argc=<optimized out>, argv=<optimized out>) at /usr/local/src/pkg/src/main.c:3213
 0(13170) INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized
 0(13170) DEBUG: <core> [core/mem/shm.c:287]: shm_destroy_manager(): destroying memory manager: q_malloc
 0(13170) DEBUG: <core> [core/mem/q_malloc.c:1278]: qm_shm_lock_destroy(): destroying the shared memory lock
 0(13170) DEBUG: <core> [core/mem/pkg.c:95]: pkg_destroy_manager(): destroying memory manager: q_malloc
[Inferior 1 (process 13170) exited normally]
Thread-specific breakpoint 2 deleted - thread 1 no longer in the thread list.
(gdb)

[space88man]

Not much information there...
Take 2 - what modules do you load?

[xadhoom]

yes, that break reports only shutdown routines.

Well actually a lot of modules:

loadmodule "tls.so"
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "sl.so"
loadmodule "db_postgres.so"
loadmodule "rr.so"
loadmodule "pv.so"
loadmodule "dialog.so"
loadmodule "maxfwd.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "textops.so"
loadmodule "textopsx.so"
loadmodule "siputils.so"
loadmodule "dmq.so"
loadmodule "htable.so"
loadmodule "dispatcher.so"
loadmodule "ipops.so"
loadmodule "kex.so"
loadmodule "auth.so"
loadmodule "usrloc.so"
loadmodule "registrar.so"
loadmodule "nathelper.so"
loadmodule "http_async_client.so"
loadmodule "ruxc.so"
loadmodule "ctl.so"
loadmodule "json.so"
loadmodule "jansson.so"
loadmodule "presence.so"
loadmodule "presence_dialoginfo.so"
loadmodule "presence_reginfo.so"
loadmodule "pua.so"
loadmodule "pua_reginfo.so"
loadmodule "pua_dialoginfo.so"
loadmodule "debugger.so"
loadmodule "sqlops.so"
loadmodule "sdpops.so"
loadmodule "rtpengine.so"
loadmodule "corex.so"
loadmodule "timer.so"
loadmodule "rtimer.so"
loadmodule "cfgutils.so"
loadmodule "mqtt.so"
loadmodule "xhttp.so"
loadmodule "jsonrpcs.so"
loadmodule "uac.so"

[xadhoom]

Ok, further progress: my setup has enable_tls=yes in the config. removing it (which disables SIP TLS) kamailio works with mqtt too. So seems something related to concurrent startup of mqtt tls and core tls?

[xadhoom]

while disabling TLS on mqtt (let it go in plain) and keeping enable_tls=yes works too

[xadhoom]

further infos: launching with "strace -ff kamailio" which slows down things... it works. so seems a race condition?

[xadhoom]

last for today: switching to tlsa works, too. Unfortunately I'm not expert enough to perform further analysys. Let me know if I can help in some way.

[space88man]

yes, that break reports only shutdown routines.

Well actually a lot of modules:

loadmodule "tls.so"
loadmodule "tm.so"
loadmodule "tmx.so"
loadmodule "sl.so"
loadmodule "db_postgres.so"
loadmodule "rr.so"
loadmodule "pv.so"
loadmodule "dialog.so"
loadmodule "maxfwd.so"
loadmodule "xlog.so"
loadmodule "sanity.so"
loadmodule "textops.so"
loadmodule "textopsx.so"
loadmodule "siputils.so"
loadmodule "dmq.so"
loadmodule "htable.so"
loadmodule "dispatcher.so"
loadmodule "ipops.so"
loadmodule "kex.so"
loadmodule "auth.so"
loadmodule "usrloc.so"
loadmodule "registrar.so"
loadmodule "nathelper.so"
loadmodule "http_async_client.so"
loadmodule "ruxc.so"
loadmodule "ctl.so"
loadmodule "json.so"
loadmodule "jansson.so"
loadmodule "presence.so"
loadmodule "presence_dialoginfo.so"
loadmodule "presence_reginfo.so"
loadmodule "pua.so"
loadmodule "pua_reginfo.so"
loadmodule "pua_dialoginfo.so"
loadmodule "debugger.so"
loadmodule "sqlops.so"
loadmodule "sdpops.so"
loadmodule "rtpengine.so"
loadmodule "corex.so"
loadmodule "timer.so"
loadmodule "rtimer.so"
loadmodule "cfgutils.so"
loadmodule "mqtt.so"
loadmodule "xhttp.so"
loadmodule "jsonrpcs.so"
loadmodule "uac.so"

Can you attach your config too

[xadhoom]

@space88man seems that fixes in current 5.7 branch after #3765 makes it work, so I think this can be closed.