Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
runtime: map memory usage grows as it changes even though number of entries does not grow #16070
I have an issue with growing memory usage for a simple map without pointers. The map contains not small amount of entries, keys are fully random. Entries to the map are added and removed but the number of entries is on the same high level. Here is a gist reproducing the issue: Gist
When provided example is run then at the beginning heap size is less than 500MB and memory in use looks like this:
After few hours of run (~ 700 repeats) heap size is around 1GB and memory in use looks like this
As you can see memory usage grows in line 35 where entry is added to the map (https://golang.org/src/runtime/hashmap.go line 429). This is because of how map works under the hood.
Before I start study and debug how exaclty map works under the hood I decided to describe my issue here. Maybe you are already familiar with it and can provide some hints on it. This issue was produced on go1.6.2 darwin/amd64.
Yes, this is a known limitation of maps. Once overflow buckets are allocated, they are not freed (until the map grows, which doesn't happen in your test). Given lots of inserts and deletes, you'll eventually get an overflow bucket or two for each main bucket.
It would be great to figure out a way to fix this. Deleting overflow buckets is tricky in the presence of iterators.
I'm working on this now. Expect an R=go1.8 early feedback CL soon.
I'm using a slightly modified version of the original code that generates less irrelevant garbage and runs
@rhysh some folks vaguely remembered that you might have expressed an interest in working on this problem at GopherCon. If so, I'd be interested to get your take on the CL. If not, apologies for the noise.