dlrlookupd.py memory usage

[kb-mobi]

Hello,

We are using jasmin 0.9.36 installed via pip on Centos7 server

We are facing a memory issue : the memory usage for dlrlookupd.py keep increasing and we need to restart jasmin to free up memory.

If i check the log I can see a lot of message like "DLRMapNotFound: Got a DLR for an unknown message". Can it be related ?

Am I missing some configuration to free up memory ?

We are sending ~ 150/160 messages / minute with DLR set.

Any idea ?

Thank you for your help

[farirat]

Are you sending long multiparts ?

[kb-mobi]

We are sending message in Unicode encoding and some can be sent in 2 or 3 parts.

Is there a way to improve this ?

Thank you for your help.

[kb-mobi]

Hello,

We have reached the point where we need to restart jasmin again :

Any idea about this behavior ?

Thank you for your help

[magojr]

Are you sure it's due to dlrlookupd.py and not from REDIS memory usage?
What I've seen with your same OS and Jasmin version that every sms sent store it's "ID mapping" in REDIS, waiting for the DLR.
Jasmin need this info to be able to send back the DLR with it's generated ID when the one with the upstream ID is received.
If the DLR come back, the memeory should be freed, if not it's kept in memory until its timeout (that is a parameter inside the CFG file, I don't remember which one). I don't send multipart but I know a lot of operator send back DLR only for the first part (or the last) in these cases, You should have a very high usage of memory if you haven't back dlr for a lot of parts. The only way to reduce this usage is to decrease the time they will be kept in memory (by CFG file) waiting for a possoble DLR.
So if you use a timeout of 15 minutes and the user switch on his device after 1hour, you are loosing that DLR, on the other side if you keep the DLR mapping for 2days you're probably keeping them too much. For what I've seen, rarely DLR come after 8 hours and never after 12hours (even if messages can be received, queue for SMS and TTL for DLR are different).

Note that I'm speaking about REDIS memory usage (due to the DLR managment), not about dlrlookupd.py memory usage that shouldn't be so high for what I know.

[kb-mobi]

Hello @magojr and thank you for your response.

a top command clearly show dlrlookupd.py using a lot of memory.

dlr_expiry was set to 86400sec, I set it to 3600sec now to test.

[magojr]

Ok, so it may be high memory usage OR a memory leak in dlrlookupd.py. I think it shouldn't increase the memory usage so much because Redis is used for the mapping purpose and other task/data shouldn't stay in memory. I prepend for a memory leak, mabe a simple array entry not freed...
Let me know if in this way it continue to increase or if it get a stable quota. So we can understand if it's a leak or only an high usage.
Thank you!

[kb-mobi]

Here is the memory consumption after changing the settings :

Now, dlrlookupd.py memory comsumption seems OK, but dlrd.py comsumption keep increasing at a more acceptable level

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
jasmin    1799  0.0  0.2 249552 23692 ?        Ss   mars10   0:00 /usr/bin/python2 /usr/bin/interceptord.py
jasmin    1800  3.1  1.5 437368 124988 ?       Ssl  mars10  49:47 /usr/bin/python2 /usr/bin/jasmind.py
jasmin    1801  1.1  2.8 458832 224456 ?       Ss   mars10  18:10 /usr/bin/python2 /usr/bin/dlrlookupd.py
jasmin    1802  1.2  5.9 1447824 475420 ?      Ssl  mars10  19:24 /usr/bin/python2 /usr/bin/dlrd.py

[magojr]

It seems to be an increasing line, not a "curve" with a limit so it may increase up ti 7G in the same way of the previous execution.
How much time passed to achieve 7G?
If i'm not missing something you should have only a lower amount of memory usage from redis, not from any py script unless the data is stored both in memory inside the daemon and in redis too that should be a nonsense approach...
I continue to think to a memory leak, it probably happens with concatenated SMS because I didn't find an high memory usage. In the last months we are not sending high traffic, anyway the daemons are in execution since 5/6 months and consumes less than 50MB each...

[kb-mobi]

Here is the graph for the last 2 months :

Here you can see the redis, not using lots of RAM :

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
redis      938  0.1  0.0 143056  6076 ?        Ssl  janv.21 127:15 /usr/bin/redis-server 127.0.0.1:6379
jasmin    1799  0.0  0.2 249552 23692 ?        Ss   mars10   0:00 /usr/bin/python2 /usr/bin/interceptord.py
jasmin    1800  3.1  1.5 437368 124988 ?       Ssl  mars10  51:59 /usr/bin/python2 /usr/bin/jasmind.py
jasmin    1801  1.1  2.8 464768 230760 ?       Ss   mars10  18:57 /usr/bin/python2 /usr/bin/dlrlookupd.py
jasmin    1802  1.2  5.9 1447824 475420 ?      Ssl  mars10  20:18 /usr/bin/python2 /usr/bin/dlrd.py

We thought about a memory leak too, but i can't find any similar issue and i can't think we are the only ones with this issue...

[github-actions]

Stale issue detected, remove stale label or comment or this will be closed soon.

[github-actions]

This issue was closed because it has been stalled more than a year with no activity.

[hadpro24]

@kbadey I got same issue... do you find any solution ?

[kb-mobi]

@hadpro24 Unfortunatly, the issue is still present and we were unable to find a solution.

We tried to upgrade Jasmin version.We are now using the 0.10.12 version but the result is the same. We have to restart jasmin processes to free the memory. dlrlookupd.py is still using more and more memory.