Add L0 memory growth test

[jbkyang-nvi]

No description provided.

[jbkyang-nvi]

based on @dyastremsky : #3827
dyas-memory-growth...kyang-memory-growth
and @jackyh 's PR: #4054

[dyastremsky]

Beautiful! This passed for 1M iterations for me with the memory usage bouncing around within a few decimal points of 17MB. Currently running it for 20M Iterations, which will likely run for a while.

Added two small comments. Let me know what you think. You can make the updates and I can review this ticket or I can do it and we can get someone else to review this test.

[dyastremsky]

We should probably make Check only run based on a command line boolean. That way we can break up the test into a sanity test to make sure that the full Simple example has no memory leak (run for a small number of iterations) and a high-iteration memory leak test just for the inference. Right now, I suspect Check is likely taking up a lot of the time per inference, which I suspect we don't want. What do you think?

[dyastremsky]

Let's either change the previous line to only copy the simple model from models, or make this line remove L0_simple_ensemble. It's not part of the test anymore, so we don't need to load it.

[dyastremsky]

Amazing work on the PR! Heads up that the test failed on 20M iterations due to a larger memory allocation range (it's saying a 7.7MB, 32% diff). There are two parts to this:

1. The memory oscillates in the 14-17MB range. The current 10% tolerance should be good, though we can increase this to 20% if we want to ensure there's no intermittence on this test.
2. Outliers. Out of 6,750 memory readings, 3 were higher (22-23MB). I'll look into whether DoubleSummaryStatistics can provide percentiles or get rid of outliers; if not, whether there's a better way to track memory statistics.

[dyastremsky]

Amazing work on the PR! Heads up that the test failed on 20M iterations due to a larger memory allocation range (it's saying a 7.7MB, 32% diff). There are two parts to this:

1. The memory oscillates in the 14-17MB range. The current 10% tolerance should be good, though we can increase this to 20% if we want to ensure there's no intermittence on this test.
2. Outliers. Out of 6,750 memory readings, 3 were higher (22-23MB). I'll look into whether DoubleSummaryStatistics can provide percentiles or get rid of outliers; if not, whether there's a better way to track memory statistics.

I don't see a way to fix #2 with DoubleSummaryStatistics. The ideal would be to use a percentile (e.g. comparing the 90th percentile value versus the median to get the difference), but that'd be hard to do without holding all our data in a data structure, which we don't want to do (especially for the long-running memory growth test). We need to figure out a way to identify and reject/ignore outliers.