-
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test(python): Memory usage test infrastructure, plus a test for #15098 #15285
test(python): Memory usage test infrastructure, plus a test for #15098 #15285
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #15285 +/- ##
=======================================
Coverage 81.36% 81.36%
=======================================
Files 1364 1365 +1
Lines 176612 176634 +22
Branches 2525 2525
=======================================
+ Hits 143694 143716 +22
Misses 32434 32434
Partials 484 484 ☔ View full report in Codecov by Sentry. |
This is cool stuff! |
The lack of abi3 support for the tracemalloc API is apparently a problem on Windows 😢 |
It looks like (from looking at |
Asked pyo3 people with some help on the Windows linking errors, will hopefully have reply by tomorrow. |
This reverts commit 9df2adc.
@stinodego OK it's ready I think. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm pretty amazed that you've managed to set this up with so little code. Nice!
I left some minor comments on the organization of things on the Python side. And a few just for my own understanding.
#[cfg(all(target_family = "unix", debug_assertions))] | ||
static ALLOC: TracemallocAllocator<Jemalloc> = TracemallocAllocator::new(Jemalloc); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So debug builds will now always use the tracemalloc, correct? Can we expect any impact on performance, or is it 'harmless'?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would be surprised if it has noticeable impact. In particular, it will involve an extra function call per allocation/deallocation, but unless tracemalloc is started, that function call won't do anything and will just return, so it'd be a pretty fast function. And I would expect in general Polars does not do a particularly large number of allocations, since most allocations would be large chunks.
Co-authored-by: Stijn de Gooijer <stijndegooijer@gmail.com>
…st-infrastructure
@stinodego back to you. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All right, I'm fine merging this as-is. Letting @ritchie46 make the final call, in case I missed something here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great! Thank you @itamarst
Fixes #15231
This relies on the
tracemalloc
infrastucture built-in to Python. It is not supported by all libraries (notably PyArrow) but it is cross-platform, unlike other alternatives.Followups: