kamailio tls connection uses too much shared memory

[JiangHai2011]

Description

a tls connection uses 52104 bytes. Among these memory, tcp_connection structure use 776 bytes and tcp_rd_buf use 6000 bytes, and the left part (45328 bytes) are all about SSL session with crypto.

Expected behavior

kamailio does some optimization for self defined BIO_TYPE_SOURCE_SINK bio type, to save more memory

Actual observed behavior

among this 45328 bytes, the biggest parts are BIO read buffer(16KB) and BIO write buffer(16KB). currently kamailio uses BIO_TYPE_SOURCE_SINK type bio, which needs kamailio manage the buffer by itself (there is no optimization). While nginx uses BIO_TYPE_MEM type bio, which is a openssl's internal bio type with memory optimization. As a result, nginx use less memory to accept more TLS connections than kamailio

Debugging Data

None

Log Messages

Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1219]: tlsf_sums(): pool (0x7f1a3eec1000) summarizing all alloc'ed. fragments:
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     2 size=       336 bytes from tls: tls_init.c: crypto/evp/evp_enc.c(43)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=        80 bytes from tls: tls_init.c: crypto/bn/bn_blind.c(36)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     3 size=       360 bytes from tls: tls_init.c: crypto/bn/bn_mont.c(232)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     2 size=      1456 bytes from tls: tls_init.c: crypto/evp/evp_enc.c(129)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=       120 bytes from tls: tls_init.c: ssl/t1_lib.c(1784)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=        56 bytes from tls: tls_init.c: ssl/statem/extensions.c(959)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     2 size=       112 bytes from tls: tls_init.c: ssl/t1_lib.c(1811)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=        56 bytes from tls: tls_init.c: ssl/statem/../packet_local.h(462)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=        56 bytes from tls: tls_init.c: ssl/statem/../packet_local.h(485)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=       640 bytes from tls: tls_init.c: ssl/ssl_sess.c(72)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=       144 bytes from tls: tls_init.c: ssl/packet_local.h(462)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=       224 bytes from tls: tls_init.c: crypto/evp/digest.c(139)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     3 size=       168 bytes from tls: tls_init.c: crypto/evp/digest.c(62)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=     16496 bytes from tls: tls_init.c: ssl/record/ssl3_buffer.c(124)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=     16712 bytes from tls: tls_init.c: ssl/record/ssl3_buffer.c(63)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=      6280 bytes from tls: tls_init.c: ssl/ssl_lib.c(691)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=      6776 bytes from core: tcp_main.c: tcpconn_new(1148)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=        56 bytes from tls: tls_init.c: tls_bio.c(184)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=       120 bytes from tls: tls_init.c: crypto/bio/bio_lib.c(73)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=       536 bytes from tls: tls_init.c: ssl/ssl_cert.c(76)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=        56 bytes from tls: tls_init.c: ssl/ssl_lib.c(793)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=        72 bytes from tls: tls_init.c: crypto/bio/bio_meth.c(41)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=        96 bytes from tls: tls_init.c: crypto/bio/bio_meth.c(38)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=      1040 bytes from tls: tls_init.c: ssl/s3_lib.c(3296)
Mar 30 19:46:46 localhost.localdomain sipproxy[2273]: INFO: <core> [mem/tlsf_malloc.c:1235]: tlsf_sums():  count=     1 size=        56 bytes from tls: tls_server.c: tls_complete_init(229)

SIP Traffic

None

Possible Solutions

None

Additional Information

[root@ip-10-23-0-191 ec2-user]# /opt/kamailio/sbin/kamailio -v
version: kamailio 4.4.7 (aarch64/linux) 
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT-NOSMP, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown 
compiled on 12:18:05 Mar 28 2024 with gcc 7.3.1

• Operating System:

Linux localhost.localdomain 4.18.0-425.3.1.el8.x86_64 #1 SMP Tue Nov 8 14:08:25 EST 2022 x86_64 x86_64 x86_64 GNU/Linux

[miconda]

TLS is known to be greedy in memory, depending also on the encryption algorithm negotiated. Also, kamailio does many times speed optimisations at the expense of some memory (e.g., static buffers or allocated at startup to avoid often alloc/dealoc at runtime).

If you think there is room for improvement here, on this particular case, feel free to make a PR and if the results are good overall, then it will be merged.

[github-actions]

This issue is stale because it has been open 6 weeks with no activity. Remove stale label or comment or this will be closed in 2 weeks.