jemalloc "extension" for Linux #4971
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…e errors that came from renaming to files to .cpp
|
Same issue after changing to your approach. Seems like intptr_max == int64_max despite being 32bit? Or I am misunderstanding something. Might need to switch to the env variable approach |
|
It could be that the crash is happening in the jemalloc start-up code or so, which might still break even if none of the code is actually used as long as it is compiled in. Perhaps excluding the code entirely is a better idea indeed |
|
All green! @Mytherin |
|
Awesome! Thanks for all the hard work :) |
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the jemalloc "extension", that we will for 64-bit Linux builds. The default allocator on Linux creates memory fragmentation, which eventually causes allocations to fail (see #3969). jemalloc should do a better job at this. We do not use jemalloc as a global allocator, only our
Allocatorclass calls into jemalloc.I've put "extension" between quotation marks, because jemalloc can only be loaded statically (no
INSTALL/LOADpossible). We've chosen to do this because switching between allocators dynamically causes a bunch of issues with blocks that are allocated while the switch happens.This was much more work than expected, because I've converted jemalloc to C++, and wrapped it in a namespace to prevent conflicts. jemalloc also has a bunch of platform-specific optimizations, which I had to rip out to make it portable.
Somehow this has caused a regression in TPC-DS q72 in my CI runs, which I cannot reproduce locally. Not sure what is going on there.
Happy to receive any feedback!
P.S.: jemalloc works on macos too, but not (yet) on windows, but we will probably not include it on macos since the default allocator is OK