Memory usage increases everytime tls.reload is executed

[denzs]

Description

We are using Kamailio 5.7.4 on Debian 12 (from http://deb.kamailio.org/kamailio57) with rtpengine as an Edgeproxy for our clients. The instance terminates SIP/TLS (with Cliencertificates) and forwards the SIP Traffic to internal systems.

After some days we are getting errors like this
tls_complete_init(): tls: ssl bug #1491 workaround: not enough memory for safe operation: shm=7318616 threshold1=8912896

First we thought Kamailio just doesnt have enough memory, so we doubled it..

But after some days the Logmessage (and Userissues) occured again.

So we monitored the shmmem statistics and found that used and max_used are constantly growing til it reaches the limit.

As i mentioned we are using client-certificates and so we are also using the CRL feature.
We do have a systemd-timer which fetches the CRL every hour and runs 'kamcmd tls.reload' when finished.

Our tls.cfg looks like this:

[server:default]
method = TLSv1.2+
private_key = /etc/letsencrypt/live/hostname.de/privkey.pem
certificate = /etc/letsencrypt/live/hostname.de/fullchain.pem
ca_list = /etc/kamailio/ca_list.pem
ca_path = /etc/kamailio/ca_list.pem
crl = /etc/kamailio/combined.crl.pem
verify_certificate = yes
require_certificate = yes

[client:default]
verify_certificate = yes
require_certificate = yes

After testing a bit we found that every time tls.reload is executed Kamailio consumes a bit more memory which eventually leads to all the memory being consumed which leads to issues for our users.

See following example:

[0][root@edgar-dev:~]# while true ; do /usr/sbin/kamcmd tls.reload ; /usr/sbin/kamcmd core.shmmem ; sleep 1 ; done
Ok. TLS configuration reloaded.
{
	total: 268435456
	free: 223001520
	used: 41352552
	real_used: 45433936
	max_used: 45445968
	fragments: 73
}
Ok. TLS configuration reloaded.
{
	total: 268435456
	free: 222377960
	used: 41975592
	real_used: 46057496
	max_used: 46069232
	fragments: 78
}
Ok. TLS configuration reloaded.
{
	total: 268435456
	free: 221748664
	used: 42604992
	real_used: 46686792
	max_used: 46698080
	fragments: 77
}
Ok. TLS configuration reloaded.
{
	total: 268435456
	free: 221110832
	used: 43242408
	real_used: 47324624
	max_used: 47335608
	fragments: 81
}
^C
[130][root@edgar-dev:~]# 

Troubleshooting

Reproduction

Everytime tls.reload is called the memory consumptions grows..

Debugging Data

If you let me know what would be interesting for tracking this down, i am happy to provide logs/debugging data!

Log Messages

If you let me know what would be interesting for tracking this down, i am happy to provide logs/debugging data!

SIP Traffic

SIP doesnt seem to be relevant here

Possible Solutions

Calling tls.reload less often or restart kamailio before memory is consumed ;)

Additional Information

version: kamailio 5.7.4 (x86_64/linux) 
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown 
compiled with gcc 12.2.0

• Operating System:

* Debian GNU/Linux 12 (bookworm)
* Linux edgar-dev 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux