Account for the old generation in memory measurements #216
Conversation
The erlang gc information seems to report old and young generation separately. Here is an example from faulty measurements we used: ```elixir [ old_heap_block_size: 0, heap_block_size: 610, mbuf_size: 77, recent_size: 0, stack_size: 14, old_heap_size: 0, # <--- notice me heap_size: 540, bin_vheap_size: 0, bin_vheap_block_size: 46422, bin_old_vheap_size: 0, bin_old_vheap_block_size: 46422 ] [ old_heap_block_size: 2586, heap_block_size: 1598, mbuf_size: 0, recent_size: 198, stack_size: 14, old_heap_size: 355, # <--- notice me heap_size: 275, bin_vheap_size: 0, bin_vheap_block_size: 46422, bin_old_vheap_size: 0, bin_old_vheap_block_size: 46422 ] ``` Notice how `heap_size` got smaller but together with the `old_heap_size` memory was still consumed? Yup, that's what we're talking about here. So, always take both of them into account.
|
yeah sadly couldn't think of a way to test this... The only "test" I can think of is to inline Michals benchmarking suite and run an integration test against it... wanted to do that either way but I need both PRs merged for that. Will try that once both are merged :) Thanks for the review 👍 |
|
Even that test will be tricky because of variation in memory usage between environments. I was seeing pretty significant differences between my local machine and CI. I guess we could run that benchmark and at least ensure that none of the measurements come back negative? |
|
@devonestes Oh just saw this. So, because the measurements were always the same and the new output code puts out N/A and the warning if things go horribly wrong we can test for their absence. So that's good. For me, it also seemed to work when eliminating all the other scenarios - so maybe even a unit test with these in the memory module would work... intriguing. See #218 |
The erlang gc information seems to report old and young generation
separately.
Here is an example from faulty measurements we used:
Notice how
heap_sizegot smaller but together with theold_heap_sizememory was still consumed?Yup, that's what we're talking about here.
So, always take both of them into account.
Solves #213 when combined with #214 - wanted to split it out because one is about handling faulty values in statistics/formatting and this one is nice and separate in the memory measurement department.