Avoid tracking small allocations by setting memThreshold
#935
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In order to avoid running out of memory and crashing the server, Arkouda
uses
memTrack
to track memory allocations and raise a client errorwhen a large allocation would likely exceed the available memory.
Recently, we discovered some performance overheads from this tracking
while doing highly concurrent short-lived allocations. The
memTrack
flag is implemented with a global lock to protect access to a hash
table, which is serializing and really hurting the performance of
concurrent allocs. To limit the impact of this, use
memThreshold=1M
toonly track larger allocations and skip skip tracking smaller ones.
Using
memThreshold
did not have quite the performance boost I wasexpecting at first with Chapel 1.25 because we didn't have a way to do
threshold checks on free. That was added in chapel-lang/chapel#18465.
Here's an allocation micro-benchmark that shows the performance on a
single node with 128-Rome CPUs:
So this should benefit code that has highly concurrent allocations like
casting to strings and currently regex operations (though we're working
on optimizing the regex allocations out.)
Part of #675
Part of #929