Implement a rough memory Limit for the IdTables #344
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
joka921
force-pushed
the
f.memoryLimit
branch
4 times, most recently
from
a8dd5d9
to
769ea70
Compare
hannahbast
reviewed
Mar 23, 2021
There was a problem hiding this comment.
Thank you for this valuable PR which saved us from hundreds of avoidable QLever crashes already. We already discussed much of it via ZOOM. Most of my comments are about better names, comments, or logging. Note that several of my comments have the property that they have consequences for the names in a lot of places, but I only made the comment once.
Let me know if you don't agree with a name change or if it seems that I misunderstood something.
hannahbast
requested changes
Mar 29, 2021
- It keeps track of the number of maximally allowed bytes, using a synchronized, shared state, checks if an allocation is OK. - The actual allocation is still done by a std::allocator. - The IdTable now also internally uses an allocator template insteead of malloc and free. - Integrated the memory Limit into the complete project. - Required minor adaptations for a lot of Unit tests.
hannahbast
approved these changes
Mar 29, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Allows limiting the size of all IdTables that are used at the same time by using a stateful allocator.
Exceeding the limit will cause an exception.
We currently have no automatic cleanup mechanism etc. , clearing the cache
is the only possibility to get out of the OOM state under heavy load.
However, this is very useful nonetheless because it prevents most
std::bad_alloc(or even worse OOM or silent)crashes.
TODO: