Memory leak in Cardano transactions signature/proof

[jpraynaud]

Why

There is a memory leak in the signature/proof generation of the Cardano transactions. A first investigation shows that it seems to be due to the Merkle tree used to sign that is not released from memory (~2GB leak on testing-preview)

What

Investigate and fix the memory leak

How

• Deactivate the signature of Cardano transactions on testing-preview
• Investigate the memory leak
• Fix the problem
• Reactivate the signature of Cardano transactions on testing-preview

[jpraynaud]

Analysis

Memory leak analysis

The investigations have led us to identify that the problem is occurring when the Cardano transactions is activated and that it is probably due to the Merkle tree used to sign.

We have created a test to identify the memory behavior with this Merkle tree:

• Create the data used to feed to Merkle tree
• Wait
• Loop 3 times:
  • Load the Merkle tree
  • Drop the Merkle tree
  • Wait

We have implemented Drop methods on all the structs of the corresponding module and we have confirmed that all the objects created were properly dropped within the test which indicates that there is no reference counting issue and no cycle created, i.e no memory leak

We have profiled the test with 10M leaves and have confirmed that there is no memory leak, and the following memory profiled was created:

Memory fragmentation

After investigation, it appears that the RES memory was being released properly by the program. With the same test as above, we had recorded this process memory evolution (where memory is never given back to the OS):

Fix

We are currently using HashMap and BtreeMap which tend to create memory fragmentation when the number of elements is high. The fix consist in replacing the default memory allocator of the program in order to avoid memory fragmentation. We have implemented it in PR #1649 with the jemallocator memory allocator. We have re-run the same test as above and see that the memory is now given back to the system properly: