Severe memory leak detected

[T-Shilov]

Hi Per,

Unfortunately, in the process of using TrackDirect, a serious memory leak was discovered.
RAM or swap partition for 6-7 days overflows every time and the system collapses.
I've tried different combinations of swap partition volume and swappiness ratio 0 to 60, but it doesn't help.
So far, I have temporarily switched to a daily reboot of the system, but this is not the best solution.

Per, I very hope you will solve this annoying problem.

[qvarforth]

I have been running this code for several years and have never seen this problem (one of my current servers has been up for 64 days).

Do you have the possibility to add information about which process that eats all the memory? (current data applies to the entire virtual server which means it could be anything)

[T-Shilov]

Hello Per,

I installed the smem utility this morning and am monitoring the memory behavior.
It's too early to say for sure, but we can assume that this process is to blame:

python2 ./bin/collector.py trackdirect.ini 0

Because in the morning he "ate" 314.9M:

2022.04.05 10:20:01

----------------------------------------------------------
PID User     Command      Swap      USS      PSS      RSS
835 aprsmap  python2 ./bin/collector.py   0 314.9M   315.2M   320.8M

and in 2022.04.05 19:25:01 it became more: 464.1M

----------------------------------------------------------
PID User     Command    Swap      USS      PSS      RSS
 835 aprsmap  python2 ./bin/collector.py  0  464.1M   464.4M   470.0M

And it probably continues to grow further.

I will continue to observe and inform you about the behavior of this process.

[qvarforth]

Hi,

Thank you for investigating the problem.

The use of 300 Mb by the collector seems pretty reasonable, it has a in memory database to handle the the frequency limit (to reduce cpu and database usage). But if it grows much larger in just a day or two we should investigate it.

I'm waiting for your update.

[T-Shilov]

Hi,

April, 6
It became even more: 1.0 GB

[T-Shilov]

Hi,

Continue:

1.6 GB

[T-Shilov]

Hi,

Continue:

2.3 GB

[qvarforth]

The collector has a lot in memory to handle the frequency limit (to avoid having to always fetch to much from db). The intention was never for this data to be this large. I will try to reduce the amount of data that we keep in memory.

[T-Shilov]

Hi,

Continue:

2.5 GB

The swap is no longer zero, as before, but 414 MB.
A little more, and there will be a collapse, as here:
#18 (comment)
Therefore, I have to turn on the regular reboot of the system again :-(

[qvarforth]

Hi,
I have now improved memory handling for the frequency limit functionality, I have also added the possibility to disable the duplicate detection functionality (which uses some memory). Please try the updated code. If you still see a high memory usage, disable the frequency limit (set collector.frequency_limit to "0") and disable duplicate detection (set collector.detect_duplicates to "0").

[T-Shilov]

Thank you very much, Per!

I am testing your improvement soon, and will inform you.

[T-Shilov]

Hi,

1. collector.py behavior after code update

Immediately after the start of the application: 34.7 MB

2. Per, let me ask you one more question: how to launch your product in a slightly different way?

For example, instead of https://example.com/ this: https://example.com/aprs/

What changes of code do me need to make to your products for this?

[T-Shilov]

Hi, Per,

Unfortunately, the memory used is growing: 1.1 GB
But I haven't followed your recommendations yet -

If you still see a high memory usage, disable the frequency limit
(set collector.frequency_limit to "0") and disable duplicate detection
(set collector.detect_duplicates to "0").

• are they mandatory?

[qvarforth]

Please try to disable the frequency limit (set collector.frequency_limit to "0") and disable duplicate detection (set collector.detect_duplicates to "0"), that will affect memory usage a lot.

It should be possible to place the website at any url (including subdirectories). But if you place it in a subdirectory the "automatic" websocket url finder will not work, you need to set "websocket_url" manually in config file.

[T-Shilov]

Thanks you Per, I'll do it.

[T-Shilov]

Hello Per,

Great your job!

After I used your improvements, the memory leak stopped, and I have been observing the steady operation of your product for a month.
collector.py now uses a negligible amount of memory:

Bravo Per :-)