test(python): Memory usage test infrastructure, plus a test for #15098

[itamarst]

Fixes #15231

This relies on the tracemalloc infrastucture built-in to Python. It is not supported by all libraries (notably PyArrow) but it is cross-platform, unlike other alternatives.

Followups:

• I tried to also track PyArrow memory usage, but failed. Possibly because it's hard, possibly because there's a memory leak somewhere... I will try to poke at this some more.
• Once this is merged, there are probably a bunch of other code paths that could benefit from this testing infrastructure.

[codecov]

Codecov Report

Attention: Patch coverage is 86.36364% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 81.36%. Comparing base (03c5f73) to head (2cb081d).

Files	Patch %	Lines
py-polars/src/memory.rs	86.36%	3 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #15285   +/-   ##
=======================================
  Coverage   81.36%   81.36%           
=======================================
  Files        1364     1365    +1     
  Lines      176612   176634   +22     
  Branches     2525     2525           
=======================================
+ Hits       143694   143716   +22     
  Misses      32434    32434           
  Partials      484      484           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ritchie46]

This is cool stuff!

[itamarst]

The lack of abi3 support for the tracemalloc API is apparently a problem on Windows 😢

[itamarst]

It looks like (from looking at pyo3-ffi) that I need to specify the DLL link name on Windows, so going to try that. If I can't make that work, I will just skip these tests on Windows.

[itamarst]

Asked pyo3 people with some help on the Windows linking errors, will hopefully have reply by tomorrow.

[itamarst]

@stinodego OK it's ready I think.

[stinodego]

I'm pretty amazed that you've managed to set this up with so little code. Nice!

I left some minor comments on the organization of things on the Python side. And a few just for my own understanding.

[stinodego]

So debug builds will now always use the tracemalloc, correct? Can we expect any impact on performance, or is it 'harmless'?

[itamarst]

I would be surprised if it has noticeable impact. In particular, it will involve an extra function call per allocation/deallocation, but unless tracemalloc is started, that function call won't do anything and will just return, so it'd be a pretty fast function. And I would expect in general Polars does not do a particularly large number of allocations, since most allocations would be large chunks.

[itamarst]

@stinodego back to you.

[stinodego]

All right, I'm fine merging this as-is. Letting @ritchie46 make the final call, in case I missed something here.

[ritchie46]

Looks great! Thank you @itamarst