[BUG]: Memory Leak

[AlmostInteractive]

Describe the bug
Activating the iCal sensor node leaks memory until eventually it crashes HA.

To Reproduce
This doesn't seem to work with all gmail ical urls. I tried with two of mine; one causes a very noticeable memory leak and the other seems to have none at all.
Steps to reproduce the behavior:

1. setup an ical sensor to a gmail account ICS (one that triggers the memory leak, not all do)
2. set it on a manual trigger that repeats (quickly for fastest results. The screenshot below was taken with a sensor triggered at 1 sec interval)
3. watch the NodeRed memory usage climb rapidly

Expected behavior
No memory leak.

Screenshots

Additional context
If it would be possible to add a unit test or something similar, I could plug in my personal ICS file and help debug the memory leak.

[AlmostInteractive]

Just another view of the memory leak. GC isn't cleaning it up.

[nescafe2002]

Hello,

I have the same issue, it seems related to the "Use cache in case of an error" setting in the calendar config.

Error in docker container:

<--- Last few GCs --->

[7:0x7f97205a43d0] 451743753 ms: Scavenge 2044.5 (2063.9) -> 2043.3 (2067.9) MB, 3.2 / 0.0 ms  (average mu = 0.965, current mu = 0.179) allocation failure
[7:0x7f97205a43d0] 451744564 ms: Mark-sweep (reduce) 2046.5 (2067.9) -> 2044.0 (2063.4) MB, 673.2 / 0.1 ms  (+ 51.5 ms in 41 steps since start of marking, biggest step 5.9 ms, walltime since start of marking 811 ms) (average mu = 0.934, current mu = 0.454

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
./entrypoint.sh: line 14:     7 Aborted                 (core dumped) /usr/local/bin/node $NODEOPTIONS node_modules/node-red/red.js --userDir /data $FLOWS

Repro:

(Inject interval every 5s, ICS is 996 kB and has many recurring appointments)

@AlmostInteractive: can you confirm you have enabled the "Use cache in case of an error" setting?

[AlmostInteractive]

@nescafe2002 I had "Use cache in case of an error" set in my sensors, yes. I've disabled that setting and ran the test again. This is the result:

Still steadily increasing.

[nescafe2002]

That's odd - are you sure you can reproduce the issue with just (only) the ical sensor?

[AlmostInteractive]

@nescafe2002 edit deleting my previous post.

Turns out one of those was an ical-events not an ical-sensor. Even though it was not using the cached version, it looks like it also has a huge memory leak somewhere. I deactivated it and started the test over. Here's what it looks like for the past half hour with just ical-sensors and no caching:

[AlmostInteractive]

@nescafe2002 So here's the workflow I'm using:

Four sensors not using caching setup to trigger every second.

[AlmostInteractive]

@nescafe2002 I made a simpler experiment. I turned off all calendar nodes and rebooted my system. I ran it from 1:30 to 4:45 with no calendar nodes. At 4:45 I created and activated this workflow:

And the result is pretty clear:

Do you have any idea how to get this into a debugger? If I can get a dev environment setup then I'd be happy to fix this myself.

[naimo84]

Hey @nescafe2002, @AlmostInteractive,

unfortunately I had no time the last few months for doing some NodeRED coding, because I was studying for the software archticeture certificate ;)
https://www.certible.com/verify/38cc4820b5e0d574baaec29b50e1fecb/
YEAH :D

As this is finished now, I will have a look at the new issues. Getting this into a debugger is a bit tricky. If there's time, I'll write a docu.

Greets,
Benjamin

[AlmostInteractive]

@naimo84 If you have time, we can connect over Discord or something. If you can show me how to get this into a debugger, I'd be happy to fix this bug and write the documentation for the repo. That might be faster for you.

[naimo84]

@AlmostInteractive Thanks for offering your help, but spare time is unpredictable with a fulltime job as a teamlead, a wife and a small daughter 😁

writing the docu was not such a big part. May you have a look at https://naimo84.github.io/node-red-contrib-ical-events/guide/debug.html and let me know if it's working for you? Thanks in advance 👍

The markdown can be found in the docu branch 😉 https://github.com/naimo84/kalender-events/blob/docs/docs/guide/debug.md

[AlmostInteractive]

@naimo84 I completely understand. 😄 Thanks for these documents, I'll try to get this into a debugger today. Expect me to come back with questions or fixes! 👍

[AlmostInteractive]

Minor update:
The jaggedness of the memory usage comes from extendEvent but the memory leak is somewhere else. That all eventually gets cleaned up.

I've completely erased all functionality from the ical-sensor node apart from creating a node and registering it. Literally only this:

module.exports = function (RED: any) {
  function sensorNode(config: any) {
    RED.nodes.createNode(this, config);
  }
  RED.nodes.registerType('ical-sensor', sensorNode);
};

and I still get a weird memory graph:

It's slow, but it's consistent. And it is different from having that same trigger node going to the debug node which results in no memory leak, so that's the baseline.

[AlmostInteractive]

@naimo84 The PR is up with the results in the comments.
#162

[AlmostInteractive]

@naimo84 Thanks for your support. I learned quite a lot hunting down that memory leak, enough that I decided to write my own calendar nodes based on your work. It's 99% your work trimmed down and rewritten to be exactly what I need for my project:
https://github.com/AlmostInteractive/node-red-contrib-basic-ical-calendar

I gave you credit but if there's any information you'd like added, removed, or changed then I'm happy to do that.

[naimo84]

@naimo84 Thanks for your support. I learned quite a lot hunting down that memory leak, enough that I decided to write my own calendar nodes based on your work. It's 99% your work trimmed down and rewritten to be exactly what I need for my project: https://github.com/AlmostInteractive/node-red-contrib-basic-ical-calendar

I gave you credit but if there's any information you'd like added, removed, or changed then I'm happy to do that.

Thank you so much for reaching out and sharing your experience with me. I'm thrilled to hear that my work on hunting down the memory leak was helpful to you. It's fantastic to see how you've taken that knowledge and built your own calendar nodes based on my work.

I took a look at your GitHub repository, and I must say, it's impressive! I appreciate that you've given credit where it's due and acknowledged my contribution to your project. As of now, everything looks great, and I'm honored to be part of your work.

If there are any specific details or information you would like me to provide, please let me know. I'm more than happy to assist you in any way I can. Feel free to reach out if you need further support or have any questions along the way.

Keep up the fantastic work, and I wish you continued success with your project!