Release 1.1.8 and 1.2.0beta7 (updated with 1.2.0beta8): possible memory leak (or not)

[scarletbright]

Hello everyone!

In attempt to trace regular sigterms, I've attached heaptrack to my mina nodes.
I would like to share the results here, hopefully they can be useful. I'm not sure the reported memory leak is actually an issue, so yes... just sharing my observations.

So, yesterday my stable (1.1.8) node core dumped after being up for 9h 42mins

systemd log:

Oct 02 21:20:28 outerspace systemd[1625]: mina.service: Main process exited, code=dumped, status=11/SEGV
Oct 02 21:20:28 outerspace systemd[1625]: mina.service: Failed with result 'core-dump'.
Oct 02 21:20:43 outerspace systemd[1625]: mina.service: Service hold-off time over, scheduling restart.
Oct 02 21:20:43 outerspace systemd[1625]: mina.service: Scheduled restart job, restart counter is at 5.
Oct 02 21:20:43 outerspace systemd[1625]: Stopped Mina Daemon Service.

apport log:

ERROR: apport (pid 17757) Sat Oct  2 21:20:25 2021: called for pid 22662, signal 11, core limit 18446744073709551615, dump mode 1
ERROR: apport (pid 17757) Sat Oct  2 21:20:25 2021: ignoring implausibly big core limit, treating as unlimited
ERROR: apport (pid 17757) Sat Oct  2 21:20:25 2021: executable: /usr/local/bin/mina (command line "/usr/local/bin/mina daemon -peer-list-file /home/outer$
ERROR: apport (pid 17757) Sat Oct  2 21:20:25 2021: executable does not belong to a package, ignoring
ERROR: apport (pid 17757) Sat Oct  2 21:20:25 2021: writing core dump to core.22662 (limit: -1)

heaptrack logs:

total runtime: 34935.53s.
calls to allocation functions: 1962866026 (56185/s)
temporary memory allocations: 1648375 (47/s)
peak heap memory consumption: 4.18G
peak RSS (including heaptrack overhead): 2.96G
total memory leaked: 4.00G

Complete mina export-logs (218.2 Mb)

Complete heaptrack logs (242.9 Mb)

I also have a core dump file, but it's 3.7Gb

After the restart the node has worked for three more hours and then:

Oct 03 00:40:25 outerspace mina[17825]: Attempted to allocate 136404444074 bytesFatal error: out of memory.
Oct 03 00:40:25 outerspace systemd[1625]: mina.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 03 00:40:25 outerspace systemd[1625]: mina.service: Failed with result 'exit-code'.
Oct 03 00:40:40 outerspace systemd[1625]: mina.service: Service hold-off time over, scheduling restart.
Oct 03 00:40:40 outerspace systemd[1625]: mina.service: Scheduled restart job, restart counter is at 6.
Oct 03 00:40:40 outerspace systemd[1625]: Stopped Mina Daemon Service.

heaptrack log this time:

total runtime: 11615.32s.
calls to allocation functions: 635153251 (54682/s)
temporary memory allocations: 1118401 (96/s)
peak heap memory consumption: 1.93G
peak RSS (including heaptrack overhead): 2.75G
total memory leaked: 1.83G

Complete heaptrack log (88.6 Mb)

Now the node is up for 12+ hours.

System info:
Ubuntu 18.04.6 LTS
8c/16t i9-11900K @ 3.50GHz
32Gb RAM
mina 1.1.8 running BP + snark worker

---

And what about 1.2.0beta7?

Well, it's currently up for almost 20 hours. Currently heaptrack is saying:

total runtime: 70725.94s.
calls to allocation functions: 1517358254 (21454/s)
temporary memory allocations: 530319 (7/s)
peak heap memory consumption: 7.44G
peak RSS (including heaptrack overhead): 26.50G
total memory leaked: **7.27G**

last five exit codes:

Sep 29 21:56:16 frhb65434ds systemd[1256]: mina.service: Main process exited, code=exited, status=1/FAILURE
Sep 30 13:34:03 frhb65434ds systemd[1256]: mina.service: Main process exited, code=exited, status=130/n/a
Sep 30 14:25:10 frhb65434ds systemd[1256]: mina.service: Main process exited, code=exited, status=130/n/a
Oct 02 18:05:06 frhb65434ds systemd[1256]: mina.service: Main process exited, code=exited, status=130/n/a
Oct 02 19:40:09 frhb65434ds systemd[1256]: mina.service: Main process exited, code=exited, status=130/n/a

System info of the server running beta:
Ubuntu 18.04.6 LTS
12c/24t AMD Ryzen 9 3900X @3.8GHz
64Gb RAM
mina 1.2.0beta7 running BP only (no snark worker)

---

Again, I'm not even sure if those leaked numbers indicate or problem or not. Just sharing for what it's worth 😃
I don't think I was able to find the reason of sigterms, but well.... 😄
✌️

[scarletbright]

Some new numbers from the stable (1.1.8) version: after being up for 1d22h16min29s, the amount to "leaked" memory is 16Gb!

calls to allocation functions: 8839455270 (53061/s)
temporary memory allocations: 24102256 (144/s)
peak heap memory consumption: 17.52G
peak RSS (including heaptrack overhead): 5.30G
total memory leaked: 16.04G

Oct 03 00:40:25 outerspace systemd[1625]: mina.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 03 00:40:25 outerspace systemd[1625]: mina.service: Failed with result 'exit-code'.
Oct 05 06:53:54 outerspace systemd[1625]: mina.service: Main process exited, code=dumped, status=11/SEGV
Oct 05 07:58:59 outerspace systemd[1625]: mina.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 05 07:58:59 outerspace systemd[1625]: mina.service: Failed with result 'exit-code'.

[scarletbright]

Graphs from beta8 look quite different
after running for 4h42min48s:

calls to allocation functions: 1437951802 (84743/s)
temporary memory allocations: 1813623 (106/s)
peak heap memory consumption: 10.81G
peak RSS (including heaptrack overhead): 2.73G
total memory leaked: 10.53G

[joseandro]

Thanks for your detailed report @scarletbright. Could you check if the beta version is still leaking less memory than the stable version?

[scarletbright]

Yes, that's right!

Last two heaptrack reports for 1.2.0beta8:

total runtime: 24.37 hours (and still running).
calls to allocation functions: 1054863617 (12023/s)
temporary memory allocations: 3010457 (34/s)
peak heap memory consumption: 9.41G
peak RSS (including heaptrack overhead): 4.79G
total memory leaked: 9.19G

total runtime: 19,44 hours.
calls to allocation functions: 848465356 (12121/s)
temporary memory allocations: 3386705 (48/s)
peak heap memory consumption: 8.28G
peak RSS (including heaptrack overhead): 4.03G
total memory leaked: 7.68G 

---

Last two heaptrack reports for stable 1.1.8:

total runtime: 49.25 hours.
calls to allocation functions: 9686504706 (54631/s)
temporary memory allocations: 13016472 (73/s)
peak heap memory consumption: 18.39G
peak RSS (including heaptrack overhead): 5.58G
total memory leaked: 17.61G

total runtime: 46,27 hours.
calls to allocation functions: 8839455270 (53061/s)
temporary memory allocations: 24102256 (144/s)
peak heap memory consumption: 17.52G
peak RSS (including heaptrack overhead): 5.30G
total memory leaked: 16.04G