Bug: PowerShell Background daemon might still contain "memory" leaks

[LordHepipud]

Running the Icinga PowerShell Background daemon with Start-IcingaServiceCheckDaemon in background might cause memory leaks over time in case plenty of checks are configured.

Expected Behavior

A general memory consumption is expected, as we store performance metrics in memory, should how ever reach a point on where the memory stagnates.

Current Behavior

Memory might still be growing over time, even when all checks reached the limit for stored performance metrics and cleanup jobs remove no longer required data.

Possible Solution

Steps to Reproduce (for bugs)

1. Register Service Check Daemon
2. Add all available Check Commands for check execution
3. Review the memory and see it growing

[LordHepipud]

I might have found one issue in this matter. The "problem" is that registering all plugins as they are, we might run into internal exceptions because of data type errors, or arguments for plugin execution are missing.

Because of this, PowerShell keeps writing those errors into the error stack.

The goal would be to reduce these errors to a minimum, we do how ever require to keep the error stack clean.

Currently I'm testing with the following.

Modify lib\daemons\ServiceCheckDaemon\Start-IcingaServiceCheckDaemon.psm1
Line 181

Replace

} catch {
    # Todo: Add error reporting / handling
}

with

} catch {
    # Todo: Add error reporting / handling
    $Error.Clear();
}

This will ensure errors within the PowerShell session are frequently cleared in case issues appear, keeping the "stack" clean.

If anyone would join and test this modification, it would be appreciated.

In case the Icinga Framework Code Cache is enabled, please ensure to update the cache afterwards with Write-IcingaFrameworkCodeCache.

[drapiti]

Will test and let you know.

[drapiti]

I might have found one issue in this matter. The "problem" is that registering all plugins as they are, we might run into internal exceptions because of data type errors, or arguments for plugin execution are missing.

Because of this, PowerShell keeps writing those errors into the error stack.

The goal would be to reduce these errors to a minimum, we do how ever require to keep the error stack clean.

Currently I'm testing with the following.

Modify lib\daemons\ServiceCheckDaemon\Start-IcingaServiceCheckDaemon.psm1
Line 181

Replace

} catch {
    # Todo: Add error reporting / handling
}

with

} catch {
    # Todo: Add error reporting / handling
    $Error.Clear();
}

This will ensure errors within the PowerShell session are frequently cleared in case issues appear, keeping the "stack" clean.

If anyone would join and test this modification, it would be appreciated.

In case the Icinga Framework Code Cache is enabled, please ensure to update the cache afterwards with Write-IcingaFrameworkCodeCache.

I think you have found it, it looks much better.

[LordHepipud]

Thanks for the input! I will keep testing further to check if memory consumption increases over time and either add this fix to 1.5.0 or 1.4.1 already

[drapiti]

Thanks for the input! I will keep testing further to check if memory consumption increases over time and either add this fix to 1.5.0 or 1.4.1 already

So far doing some back to back tests on various servers with the same settings, the above is a big improvement but the memory consumption is still way to high, it eats it away much too quickly. There must be something else causing problems.

[LordHepipud]

I digged a little deeper on how PowerShell internally works and how shell instances are dealing with freeing memory.
It seems that sometimes PowerShell is simply "waiting" for the operating system to tell PowerShell to cleanup no longer required memory.

I now did a test by "forcing" PowerShell to cleanup no longer required memory by calling the internal garbage collector manually.

How to test this:

Modify lib\daemons\ServiceCheckDaemon\Start-IcingaServiceCheckDaemon.psm1
Search for this line:

$PassedTime += 1;
Start-Sleep -Seconds 1;

replace it with

$PassedTime += 1;
Start-Sleep -Seconds 1;
[System.GC]::Collect();

Together with the above mentioned error stack clearing, this should keep the memory on a acceptable level. By registering all checks im running around 700-750MB of memory usage for the system so far.

[LordHepipud]

I think the PR I updated just now resolves ALL issues we have with the background daemon. Took a while but now data is processed properly and only "failing" plugin calls should throw an exception which is later cleaned.

Data should no longer stay inside the memory caused by certain errors , we should clean the stack and the garbage collector is dealing with the rest.

If anyone could please test the state of the PR as of now and confirm my findings, it would be appreciated 😂

[drapiti]

I think the PR I updated just now resolves ALL issues we have with the background daemon. Took a while but now data is processed properly and only "failing" plugin calls should throw an exception which is later cleaned.

Data should no longer stay inside the memory caused by certain errors , we should clean the stack and the garbage collector is dealing with the rest.

If anyone could please test the state of the PR as of now and confirm my findings, it would be appreciated 😂

Currently testing and so far I can confirm this seems to resolve the issue. As always will check back over a few days, but the difference is very notable, so far it stays below 300Mb using the same number of checks which is a huge improvement.