You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When decode columns, we use IColumn's reserve(size_t rows) interface to avoid reallocating memories each time. However, for string columns, the element size is not fixed, and we used 64 bytes as default value. This will lead to much larger memory usage when actual element size is much smaller than 64 bytes.
This problem will be worsen for Join operator, since all build side's blocks will be in memory for a while.
The text was updated successfully, but these errors were encountered:
yibin87
changed the title
String column decode reserved much larger memory than needed
String column decoder reserved much larger memory than needed
May 6, 2023
yibin87
changed the title
String column decoder reserved much larger memory than needed
String column decoder reserves much larger memory than needed
May 6, 2023
Bug Report
When decode columns, we use IColumn's reserve(size_t rows) interface to avoid reallocating memories each time. However, for string columns, the element size is not fixed, and we used 64 bytes as default value. This will lead to much larger memory usage when actual element size is much smaller than 64 bytes.
This problem will be worsen for Join operator, since all build side's blocks will be in memory for a while.
The text was updated successfully, but these errors were encountered: