Add memory leak detection to unit tests

[dblock]

Is your feature request related to a problem? Please describe.

In #1427 a memory leak was introduced and fixed in #3390. Is there some static analysis that could have caught it in the PR?

Describe the solution you'd like

Integrate memory leak detection into unit/integration tests. For example, https://github.com/andywer/leakage.

[joshuarrrr]

Another potential tool: https://engineering.fb.com/2022/09/12/open-source/memlab/

[ashwin-pc]

@joshuarrrr Thats a really neat tool. I was thinking of a slightly different approach here. We already have a way to validate this on opensearch afaik. The way they do it is by exposing an api that reports the memory stats when it is queried. They then run a synthetic load against a test cluster and capture the stats and then analyse it to identify any memory leaks. We too have a similar api the /status endpoint. It has a pretty neat UI too. We can run a similar synthetic workload on dashboards using the same tool.

@dblock Just like OpenSearch, i think running such a workload for each PR can be quite expensive as either a unit or integ test. I think its best if we run it during builds since they are less frequent and can be more through with the synthetic workloads.

[dblock]

I think any technology and any approach that can confidently say that #3405 would have been found before release, would work for me.