Auto spill enhance #8033
Auto spill enhance #8033
Conversation
ti-chi-bot
bot
added
release-note-none
size/L
dbms/src/Common/Arena.h
Outdated
| if (resize_callback != nullptr) | ||
| { | ||
| if (!resize_callback()) | ||
| throw ResizeException("Error in arena resize"); | ||
| } |
There was a problem hiding this comment.
Will there be a performance regression?
There was a problem hiding this comment.
I think the cost is acceptable because 1. resize/addChunk is not a high frequency used function, 2. inside resize/addChunk it need allocate memory and maybe copy some existing data, which I think makes the cost of resize_callback() ignorable
There was a problem hiding this comment.
Add unlikely for exception throw path? And add random failpoint here?
There was a problem hiding this comment.
OK, I will add unlikely for it, but for failpoint, actually it is not a standalone behavior, because if this function throws error, the related partition must be marked as to spill, otherwise, it may cause wrong result. I will add failpoint to test this case in test pr.
dbms/src/Common/Arena.h
Outdated
| if (resize_callback != nullptr) | ||
| { | ||
| if (!resize_callback()) | ||
| throw ResizeException("Error in arena resize"); | ||
| } |
There was a problem hiding this comment.
Add unlikely for exception throw path? And add random failpoint here?
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: SeaRise, yibin87 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Signed-off-by: xufei <xufei@pingcap.com>
Signed-off-by: xufei <xufei@pingcap.com>
Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>
Signed-off-by: xufei <xufei@pingcap.com>
Signed-off-by: xufei <xufei@pingcap.com>
windtalker
force-pushed
the
auto_spill_enhance
branch
from
c3824a5
to
f6eaae9
Compare
ti-chi-bot
bot
added
size/XL
|
/run-all-tests |
ti-chi-bot
bot
added
size/L
|
/run-all-tests |
|
@windtalker: Your PR was out of date, I have automatically updated it for you. At the same time I will also trigger all tests for you: /run-all-tests
If the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
|
/run-unit-test |
|
/run-unit-tests |
What problem does this PR solve?
Issue Number: ref #7738
Problem Summary:
What is changed and how it works?
AutoSpillTriggersupport spill check without cooldown time if memory usage is really highQueryOperatorSpillContexts::triggerAutoSpillJoin::InsertFromBlockHashTableandArena, support check if current data is already marked as spilled when before trying to allocate more memory, currently, used inJoinspillReceivedMessageQueueis actually not memory bounded even ifmax_buffered_bytes_in_executoris setCheck List
Tests
Side effects
Documentation
Release note