-
Notifications
You must be signed in to change notification settings - Fork 24.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ES|QL: account for page overhead when calculating memory used by blocks #108347
ES|QL: account for page overhead when calculating memory used by blocks #108347
Conversation
+ RamUsageEstimator.shallowSizeOfInstance(BooleanVectorBlock.class); | ||
+ RamUsageEstimator.shallowSizeOfInstance(BooleanVectorBlock.class) | ||
// TODO: remove this if/when we account for memory used by Pages | ||
+ RamUsageEstimator.NUM_BYTES_OBJECT_ALIGNMENT; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Strictly, this should be NUM_BYTES_OBJECT_REF
, but it would need further alignment to object size, so practically it would make no difference, to the cost of one more operation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not 100% sure we should account for the refs this way, but it does seem like a reasonable solution, at least for a while.
I think we should make a constant in Block
for this rather than use NUM_BYTES_OBJECT_ALIGNMENT
- that one feels a little random. Or, if it is random, we can explain why it's the right value on the constant.
Pinging @elastic/es-analytical-engine (Team:Analytics) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks Luigi!
ES|QL accounts for memory used by single blocks, but it doesn't account for the Page.
In particular, Page.blocks array can have a significant footprint when we have thousands of columns in the result set.
Accounting for memory used by pages is difficult at this stage, so for now we'll approximate it adding some overhead to each block, assuming that a block will typically be linked to a single page.
Fixes #108104