Memory leakage in Domoticz

[SargonofAssyria]

Using: Version: 2024.3 (build 15882)
Build Hash: 0eadb55
Compile Date: 2024-01-29 11:58:42
dzVents Version: 3.1.8
Python Version: 3.7.3 (default, Oct 11 2023, 09:51:27) [GCC 8.3.0
After this https://github.com//issues/5984 memory leakage was solved there is still an unexplained leakage present.

On this graph you can see that on 3 febr. I restarted Domoticz (B.t.w. and the Raspberry) which initially started with some 70Mb.
It settled the days after to around 130Mb and stayed there. For no reason I could discover at 9 febr. it raised to 210 Mb and settled again. The 13 febr. it started rising again, now at 350 Mb and it is still rising. I used gdb again to investigate some parts of the not released memory again. It to big to look over and investigate all > 200Mb
I found that some part are only filled with 0x00 (zero;'s) and some parts are filled with multiple loglines of only a small part of the logging (less then 1 second). But every logline is repeated more then 20-30 times. And I found a lot of parts of my dzVents scripts in some parts of the memory. I know this will be a tough one to discovered where it goes wrong. You can see from the graph that Domoticz used to be very stable in Process Memory.
I wonder if others experience the same phenomena?
And if someone found the cause.

[pipiche38]

You might want to test it with 2024.4 ;-)Envoyé de mon iPhoneLe 15 févr. 2024 à 10:04, SargonofAssyria ***@***.***> a écrit : Using: Version: 2024.3 (build 15882) Build Hash: 0eadb55 Compile Date: 2024-01-29 11:58:42 dzVents Version: 3.1.8 Python Version: 3.7.3 (default, Oct 11 2023, 09:51:27) [GCC 8.3.0 After this https://github.com//issues/5984 memory leakage was solved there is still an unexplained leakage present. 15.02.2024_09.45.55_REC.png (view on web) On this graph you can see that on 3 febr. I restarted Domoticz (B.t.w. and the Raspberry) which initially started with some 70Mb. It settled the days after to around 130Mb and stayed there. For no reason I could discover at 9 febr. it raised to 210 Mb and settled again. The 13 febr. it started rising again, now at 350 Mb and it is still rising. I used gdb again to investigate some parts of the not released memory again. It to big to look over and investigate all > 200Mb I found that some part are only filled with 0x00 (zero;'s) and some parts are filled with multiple loglines of only a small part of the logging (less then 1 second). But every logline is repeated more then 20-30 times. And I found a lot of parts of my dzVents scripts in some parts of the memory. I know this will be a tough one to discovered where it goes wrong. You can see from the graph that Domoticz used to be very stable in Process Memory. I wonder if others experience the same phenomena? And if someone found the cause. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[SargonofAssyria]

I updated to 15910, so let's wait and see.

[SargonofAssyria]

Now a few days later the BIG leakage is gone, but there still remains a small leakage present.
Will try to investigate what the origine is.

[SargonofAssyria]

After more then 10 days that Domoticz is up, I found that the leakage is about 2.5 Mb in those 10 days.
So significant small to the total memory usage.
I can not pinpoint the exact case of the leakage, it might have to do with certificates (Root CA) and/or dzVents.
But which of the cause I cannot find.
So I close this issue until I find something real.

[SargonofAssyria]

I found this error. Could that have any impact on memory leakage?

2024-03-08 07:26:23.983 Error: dzVents: Error in ProcessNotificationItem: in Json::Value::resolveReference(key, end): requires objectValue
2024-03-08 07:26:23.983 Error: Lua table notification is not published. Not all sub tables are closed!

[SargonofAssyria]

Unfortunately after a little bit more then 3 weeks Domoticz became unresponsive for a few minutes that Monit has to restart Domoticz. But until then the memory usage of Domoticz did grow slowly to over 144 Mb just before the restart.

[vlietland]

Some time ago I have posted a memory leak issue on the forum: https://www.domoticz.com/forum/viewtopic.php?p=315233&hilit=memory+leak#p315233

In my case the issue is related to the amount of enabled hardware. The more hardware I enable, thus the more devices are enabled, the more mem leaks away. There seems no relationship with the type of hardware, it seems simply related to the number of devices. Could it be related to data logging?

[gizmocuz]

Sure.... But it should not need an increasing memory issue

[vlietland]

Can I perform some sort of testing to provide more debugging information? What I did sofar is:

• Disable scripting
• Disable hardware

[pipiche38]

What type of Hardware plugin are you referring ? If these are Python plugin, I do confirm that there is a memory leak every time you restart a Python plugin.
I'm not sure anything can be done here as this is probably deep into the framework itself

[vlietland]

Below the hardware types I am referring to. If I disable all hardware there are no memory leaks. As soon as I start enabling hardware the built up of memory occurs. Most leakage occurs with enabling MQTT since that enables approx 700 sensors (=domoticz devices).

[vlietland]

herewith the memory builtup:

It is solely caused by the Domoticz process(es).
As workaround I use a cronjob to release the mem, otherwise the OOM-killer does it for me:
'30 3 * * * root service domoticz.sh restart'

ps: the release today at 11:45 was caused by updating the latest beta.

[vlietland]

Hi the output is: 22634 21.8 18.6 769888 357848 ?       Ssl  03:30 225:45 /home/admin/domoticz/domoticz -daemon -www 80 -sslwww 443 reply to:

…

I saw your post on the forum. What is the output of sudo ps aux | grep domoticz | grep -v grep Because I see you have a lot of PID's

[SargonofAssyria]

Sorry about the question. I saw later that those PID's were from threads. That's why I deleted the post.
I have about 22 Hardware active. I see an increase in leakages, but not as much as you.
With the help of pmap I see that the memory leakage, but still can not figure out where it comes from.

[SargonofAssyria]

Yesterday, out of the blue, Process Memory rapidly raised to over 140Mb (my alarm limit) and I rebooted the Raspberry.

Still I did not found the reason.

[pipiche38]

Do you have automatic backup ? I found that usually this is taking some memory, but the next time it won't.

[SargonofAssyria]

I have several backups, dzVents and external scripts by cron. Those are not the initiators.
The sudden rise starts between 15:30 and 15:35
I look in the domoticz.log, but nothing special happened during that period.
I looked is syslog, messages, cron.log and others, but nothing special.
I looked at CPU_Usage, RPi Memory Usage, but nothing that happened during or around that period.
B.t.w. the drop of "Process Usage" at 21:00 hours is from the reboot of the Raspberry Pi 4B, because I was alarmed that the Process Usage came above 140Mb

[vlietland]

I have disabled auto backup a few days ago. Also for my setup it makes no difference:

I'm curious about the little dips in the mem usage. Might it be a starting point of finding the root-cause of the leak? It corresponds with the mem usage of Domoticz.

[SargonofAssyria]

You better make pictures and conclusions with the "Process Usage" widget, because that is only about Domoticz.
Memory Usage is about the whole Raspberry Pi Memory. Those little dips might be Linux garbage collection.

[vlietland]

Ah right, was not aware of that widget. Indeed the little dips are not present here:

[SargonofAssyria]

The story continues. After I updated Domoticz to version 15910 I ran Domoticz continuously for over 4 weeks from mid April until 19 May On the picture you can see that the increase in memory took place over the whole period. 19 May I upgraded to version 15992. There was a normal memory increase as you can expect in the beginning after the start.
Today around 11:00 the increase in memory consumption rose for no reason unexpected from 120 Mb to currently 210 Mb.
Domoticz is still working as it should with no errors. I will let it run until it get unresponsive or crashes.
I looked with pmap for the memory consumption. The only thing I notice is that the chuncks that are not released are 8192 Mb in size.

[SargonofAssyria]

The Process Memory keep growing in the same pace, and it at the moment over 400 Mb. Domoticz is still running fine.
I made a Domoticz Process coredump, which is over 750 Mb.
I try to look over that dump and see if I find anything unusual, but 750 Mb is big, so I doubt if I will find anything.

[SargonofAssyria]

Did not find anything yet. At 500 Mb I had to reboot my Raspberry Pi 4. Will see what happens.

[pipiche38]

do you use OpenWeatherMap ? I wonder if there is not something in it, as this is the only plugin I enabled since a while and more-less since them I'm getting leakage that I didn't have before

[SargonofAssyria]

I use the OpenWeatherMap as Hardware, not as plugin.
But in my case it is strange that for 4 weeks the Memory Leakage is small but consistant, but suddenly it rose much more.

[pipiche38]

Sorry I mean Hardware

[SargonofAssyria]

Today I noticed this:

I updated Domoticz twice because of the new Energy Dashboard.
In the first part from 18 June I had an unnoticed dzVents error every minute.
At the second update late 19 June I discovered the dzVents error and corrected it.
The slope of the memory increase is different, but it might be normal.

The second thing I noticed is the following.
I previous had the Ziggo email problem that was corrected, but I still was left with a Ziggo error when I wanted to attach a picture to the email. If lived with that for a long time, because I got my attachment via Telegram. Recently I changed the email server, so my problem was solved, and it looks like that (part) of the memory leakage was solved.
I looked into the source, and I am not an expert on C++ coding, but I wondered if this was not a memory leak.
From smtpclient/SMTPClient.cpp

        smtp_ctx.pDataBytes = new char[rmessage.size()];
        if (smtp_ctx.pDataBytes == nullptr)
        {
            _log.Log(LOG_ERROR, "SMTP Mailer: Out of Memory!");

            curl_easy_cleanup(curl);
            curl_slist_free_all(slist1);
            return false;
        }
        smtp_ctx.sDataLength = rmessage.size();
        memcpy(smtp_ctx.pDataBytes, rmessage.c_str(), smtp_ctx.sDataLength);

        curl_easy_setopt(curl, CURLOPT_READFUNCTION, smtp_payload_reader);
        curl_easy_setopt(curl, CURLOPT_READDATA, &smtp_ctx);

        /* provide a buffer to store errors in */
        curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, errbuf);

        /* set the error buffer as empty before performing a request */
        errbuf[0] = 0;

        ret = curl_easy_perform(curl);

        curl_easy_cleanup(curl);
        curl_slist_free_all(slist1);
        delete[] smtp_ctx.pDataBytes;

        if (ret != CURLE_OK)
        {
            _log.Log(LOG_ERROR, "SMTP Mailer: Error sending Email to: %s !", m_Recipients[0].c_str());
            size_t len = strlen(errbuf);
            _log.Log(LOG_ERROR, "libcurl: (%d) ", ret);
            if(len)
                _log.Log(LOG_ERROR, "%s%s", errbuf, ((errbuf[len - 1] != '\n') ? "\n" : ""));
            else
                _log.Log(LOG_ERROR, "%s\n", curl_easy_strerror(ret));
            return false;
        }
    }
    catch (...)
    {
        _log.Log(LOG_ERROR, "SMTP Mailer: Error sending Email to: %s !", m_Recipients[0].c_str());
        return false;
    }
    return true;

In the logging I cannot distinguish between the normal error log and the catch.
But I wonder should not there be a "delete[] smtp_ctx.pDataBytes;" in the catch too?

[gizmocuz]

I'm afraid that's not it... There can't be a crash after the allocation. all these curl calls are not giving an exception

I still made a commit for this, but I am 100% sure that memory is not allocated when there is a crash here

[SargonofAssyria]

Could you also make a little change (in text) so that I can see if it is the normal error logging or the catch?
Now they are exactly the same.

[gizmocuz]

Done!

[SargonofAssyria]

Tnx, will switch back to Ziggo with email with the attachment. Will see if it was an 'normal' error or the catch.
Will come back on this.

[SargonofAssyria]

Tested it intensely, and you are right. The error in the email is NOT coming from the catch, it's a normal error.
I could not find that those email errors are responsible for the memory leakage.
So my "quest" continues, because the memory leakage is still there.