You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Created a Go program that loads a CSV file and creates a go map from values in the CSV file. Simplified loading code looks like this:
// loadCSV is passed an instance of *os.File and an integer which is the// exact number of records(lines) in the CSV file.funcloadCSV(r io.Reader, numRowsHintuint32) (map[uint64]uint32, error) {
m:=make(map[uint64]uint32, numRowsHint)
for {
// read record from csv file and populate mapm[key] =value
}
returnm, nil
}
The new.csv file has just about 10,000 more rows than old.csv but the loaded go map takes around 50MB more memory! In my specific case, this sudden increase in memory occurs at this point:
Less dramatic increase in memory. I apologize if my expectation is unreasonable.
What did you see instead?
For maps with large number of keys, memory is overallocated by a large margin even when a size hint is given and map is always loaded such that number of keys <= size hint.
The text was updated successfully, but these errors were encountered:
prashanthpai
changed the title
Memory pre-allocation high for maps with large number of keys
runtime/map: Memory pre-allocation high for maps with large number of keys
Jul 11, 2020
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
Created a Go program that loads a CSV file and creates a go map from values in the CSV file. Simplified loading code looks like this:
CSV files:
The
new.csv
file has just about 10,000 more rows thanold.csv
but the loaded go map takes around 50MB more memory! In my specific case, this sudden increase in memory occurs at this point:General reproducer:
Memory usage plot
plot.txt
What did you expect to see?
Less dramatic increase in memory. I apologize if my expectation is unreasonable.
What did you see instead?
For maps with large number of keys, memory is overallocated by a large margin even when a size hint is given and map is always loaded such that number of keys
<=
size hint.The text was updated successfully, but these errors were encountered: