Problem with process limit at startup

[jchavanton]

Description

Kamailio won't start after pulling from master

Commit triggering the problem

git show ec15b23f25ba2502a661157ae69b4dc1db923f66
Author: Daniel-Constantin Mierla <miconda@gmail.com>
Date:   Fri Sep 1 12:00:06 2017 +0200

    core: main - don't count wtimer secondary process separately
    
    - it is registered via sr_wtimer_init()
    - fixes the extra empty entry in process table reported by Juha Heinanen

Reproduction

Always with specific configuration (I can diagnose further is needed)

Log Messages

When trying to start Kamailio

 0(26583) CRITICAL: <core> [core/pt.c:285]: fork_process(): Process limit of 13 exceeded. Will simulate fork fail.
 0(26583) CRITICAL: <core> [main.c:1720]: main_loop(): cannot fork tcp main process: Invalid argument

Possible Solutions

Not sure is the config I am using is not supported or if Kamailio code should be fixed.
rolling back the commit is fixing the issue
git revert ec15b23f25ba2502a661157ae69b4dc1db923f66

[miconda]

Can you list here the names of the modules you loaded in your config? Likely there is one that creates/forks an extra process, but it doesn't declare it.

[miconda]

@jchavanton -- were you able to look more into this one as per my previous comment?

[jchavanton]

sorry for the delay, I was thinking I would receive an email notification

14 processes

/ # ps | grep kamailio
    7 root       0:03 /usr/sbin/kamailio
   24 root       0:00 /usr/sbin/kamailio
   25 root       0:00 /usr/sbin/kamailio
   26 root       0:00 /usr/sbin/kamailio
   27 root       0:00 /usr/sbin/kamailio
   28 root       0:00 /usr/sbin/kamailio
   29 root       0:00 /usr/sbin/kamailio
   30 root       0:00 /usr/sbin/kamailio
   31 root       0:00 /usr/sbin/kamailio
   32 root       0:00 /usr/sbin/kamailio
   33 root       0:00 /usr/sbin/kamailio
   34 root       0:00 /usr/sbin/kamailio
   35 root       0:00 /usr/sbin/kamailio
   36 root       0:00 /usr/sbin/kamailio

/ # kamcmd ps
7	main process - attendant
24	udp receiver child=0 sock=117.75.69.33:5060
25	udp receiver child=0 sock=117.75.69.33:5160
26	slow timer
27	timer
28	secondary timer
29	JSONRPC-S FIFO
30	JSONRPC-S DATAGRAM
31	ctl handler
32	RTIMER EXEC child=0 timer=tmetrics
33	HTable Timer
34	Dialog Clean Timer
35	tcp receiver (generic) child=0
36	tcp main process

If you do not find the answer spontaneously, I will troubleshoot it further (it may be easier for me since I can reproduce)

[jchavanton]

This is on Linux Alpine in docker

[miconda]

I asked for the list of kamailio modules you load in config.

[jchavanton]

I beleive this list is accurate, there was a lot of '#ifdef'

loadmodule "app_lua"
loadmodule "auth"
loadmodule "avpops"
loadmodule "cfgutils"
loadmodule "ctl"
loadmodule "db_postgres"
loadmodule "db_sqlite"
loadmodule "dialog"
loadmodule "dispatcher"
loadmodule "diversion"
loadmodule "htable"
loadmodule "http_client"
loadmodule "ipops"
loadmodule "jsonrpc-s"
loadmodule "jsonrpcs"
loadmodule "kex"
loadmodule "maxfwd"
loadmodule "mqueue"
loadmodule "nathelper"
loadmodule "ndb_redis"
loadmodule "path"
loadmodule "permissions"
loadmodule "pike"
loadmodule "pv"
loadmodule "ratelimit"
loadmodule "rr"
loadmodule "rtimer"
loadmodule "sanity"
loadmodule "siputils"
loadmodule "sl"
loadmodule "sqlops"
loadmodule "statistics"                                                                                                                                                                                     
loadmodule "statsd"
loadmodule "textops"
loadmodule "textopsx"
loadmodule "tm"
loadmodule "tmx"
loadmodule "topos_redis"
loadmodule "topos"
loadmodule "uac_redirect"
loadmodule "uac"
loadmodule "xhttp"
loadmodule "xlog"
loadmodule "xmlrpc"

[jchavanton]

I thought that since kamcmd ps was matching the amount of Kamailio running processes we could be able to find why the count was not matching.

For the last test I lowered the amount of childs

 0(21919) CRITICAL: <core> [core/pt.c:285]: fork_process(): Process limit of 13 exceeded. Will simulate fork fail.

As seen above there is 14 processes when the server is starting properly, I concluded that is must be one of the following :

/ # kamcmd ps
7	main process - attendant
24	udp receiver child=0 sock=12.75.169.33:5060
25	udp receiver child=0 sock=12.75.169.33:5160
26	slow timer
27	timer
28	secondary timer
29	JSONRPC-S FIFO
30	JSONRPC-S DATAGRAM
31	ctl handler
32	RTIMER EXEC child=0 timer=tmetrics
33	HTable Timer
34	Dialog Clean Timer
35	tcp receiver (generic) child=0
36	tcp main process

[jchavanton]

found it !
modparam("htable", "timer_procs", 0) # problem when set to 1

if(rank==PROC_MAIN) {
        if(ht_timer_procs>0) {
                for(i=0; i<ht_timer_procs; i++) {
                        if(fork_sync_timer(PROC_TIMER, "HTable Timer", 1 /*socks flag*/, 
                                        ht_timer, (void*)(long)i, ht_timer_interval)<0) {
                                LM_ERR("failed to start timer routine as process\n");
                                return -1; /* error */
                        }
                }
        }
}

[miconda]

Good catch, thanks for troubleshooting. The issue happened when asking for an extra clean up timer for hash tables but no hash table was defined with auto-expire (so the extra timer was not needed). Hopefully it is fixed with the patch referenced above.

[jchavanton]

I see, kind of a miss config in the end, thanks for the fix