Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix memory leaks in python extension code #231

Merged
merged 3 commits into from Nov 5, 2020

Conversation

dralley
Copy link
Contributor

@dralley dralley commented Nov 1, 2020

fixes: #230 (bz #1893459)

Additionally fixes a couple of other memory leaks. Completely removes the PyObject_ToStrOrNull() function which cannot possibly work without leaking memory.

There's no good way to make PyObject_ToStrOrNull() work. It requires
allocating an intermediate value, and we can't deallocate it because
the return value points into the buffer.

Need to just return the PyBytes PyObject so that it's lifetime can be
managed by the caller. Additionally, the PyObject_ToChunkedString()
function seems under-utilized, we can improve it and use it to keep the
ergonomics that we had before.
@goosemania
Copy link

goosemania commented Nov 2, 2020

Thanks, @dralley, for finding the spot and for your fix!

Just to give an idea of the level of improvement it brings.
Processing 300 000 RPMs, went from using 20GB+ of RAM to 1GB 😮 🙃

@kontura kontura self-assigned this Nov 3, 2020
@kontura
Copy link
Contributor

kontura commented Nov 5, 2020

That's amazing.

Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Memory leaks in PyObject_ToStrOrNull()
3 participants