Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs/small improvements to the rust doc landing page #4436

Conversation

SimonSchneider
Copy link
Contributor

fixes #4432

also included some other small improvements to the landing page docs as separate commits so they can be removed if they are unwanted

@github-actions github-actions bot added the rust Related to Rust Polars label Aug 16, 2022
@SimonSchneider
Copy link
Contributor Author

Regarding the jemalloc and mimalloc addition, I included it because we saw huge memory improvements when switching to jemalloc.
It resolved two major problems we saw with mimalloc, it seemed to have a much larger memory footprint and it seemed to never release memory back to the OS which in our long running applications caused memory to grow forever.

@ritchie46
Copy link
Member

It resolved two major problems we saw with mimalloc, it seemed to have a much larger memory footprint and it seemed to never release memory back to the OS which in our long running applications caused memory to grow forever.

I shared that experience, but didn't the memory saturate at one point? E.g. reaching an asymptote.

@SimonSchneider
Copy link
Contributor Author

SimonSchneider commented Aug 16, 2022

I would have though so as well but that was not the case. This graph is memory usage over a month for a 32Gb instance. Not optimal but it has not caused major headaches as memory grows slowly and we have been ok with the fact that an OOM crash and restart is a reasonable way of reclaiming memory
image
same graph over 1 day with jemalloc (Load is of course different but just the fact that memory usage is able to go down is positive)
image

@SimonSchneider
Copy link
Contributor Author

I will evaluate how jemalloc performs over a longer time period but initially it looks promising. I can report back in a month or so with some insights.

@ritchie46
Copy link
Member

I would have though so as well but that was not the case. This graph is memory usage over a month for a 32Gb instance. Not optimal but it has not caused major headaches as memory grows slowly and we have been ok with the fact that an OOM crash and restart is a reasonable way of reclaiming memory

Interesting. I wonder if we can modify the mimalloc settings to prevent this.

@ritchie46 ritchie46 merged commit f1cd3c4 into pola-rs:master Aug 16, 2022
@SimonSchneider SimonSchneider deleted the docs/small-improvements-to-the-rust-doc-landing-page branch August 16, 2022 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
rust Related to Rust Polars
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Clarification in documentation on whether polars-rust can be used directly in async or not
2 participants