New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Shrink blocks in HashJoin on consuming half of maximum memory #54584
Conversation
This is an automated comment for commit 404ea84 with description of existing statuses. It's updated for the latest CI running ✅ Click here to open a full report in a separate page Successful checks
|
156ea25
to
ed8c325
Compare
src/Interpreters/HashJoin.cpp
Outdated
@@ -879,6 +882,28 @@ bool HashJoin::addBlockToJoin(const Block & source_block_, bool check_limits) | |||
} | |||
} | |||
|
|||
auto max_total_bytes = table_join->sizeLimits().max_bytes; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But it will work only if the limit for JOIN size is defined. Let's also take max_memory_usage into account. Also, let's use the current memory usage for the query.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For comparison, see how we do it for external GROUP BY.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How to combine this two metrics?
We can compare memory tracker delta with max_memory_usage
and current_bytes_in_join
with max_bytes_in_join
separately (as I did in 0d51fc7).
Or compare max(memory_tracker_delta, current_bytes_in_join)
and min(max_memory_usage, max_bytes_in_join)
.
This will be good:
|
…unconditionally in HashJoin.
1930608
to
404ea84
Compare
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
max_bytes_in_join
)Followup #50112 (comment)
While with
SET max_bytes_in_join = '20G'
: