Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support query level automatical spilling when query memory usage exceeds threshold #7738

Open
windtalker opened this issue Jul 4, 2023 · 0 comments
Open

Support query level automatical spilling when query memory usage exceeds threshold #7738

windtalker opened this issue Jul 4, 2023 · 0 comments
Labels
type/feature-request

Comments

@windtalker
Copy link
Contributor

Feature Request

The goal is user can simply set a memory quota for a query(per TiFlash node), and when the query memory usage exceeds the memory quota, TiFlash will trigger spill automatically if the query contains memory intensive operators.
Is your feature request related to a problem? Please describe:

Describe the feature you'd like:

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:
@windtalker windtalker mentioned this issue Jul 4, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 1, 2023
@windtalker
Add OperatorSpillContext (#7741)
5de003a 
ref #7738
@windtalker windtalker mentioned this issue Aug 4, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 11, 2023
@windtalker
support Sort operator auto spill when query memory usage exceed thres…
a220251 
…hold (#7900)

ref #7738
This was referenced Aug 11, 2023
Merged
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Aug 15, 2023
@windtalker
support agg auto spill (#7932)
c7bce89 
ref #7738
@windtalker windtalker mentioned this issue Aug 18, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 18, 2023
@windtalker
Add tidb executor id in Join's log (#7964)
8cb548e 
ref #7738
JaySon-Huang pushed a commit to JaySon-Huang/tiflash that referenced this issue Aug 18, 2023
@windtalker @JaySon-Huang
Add tidb executor id in Join's log (pingcap#7964)
be77811 
ref pingcap#7738
ti-chi-bot bot pushed a commit that referenced this issue Aug 22, 2023
@windtalker
Some refine of auto spill check (#7935)
6559c82 
ref #7738
ti-chi-bot bot pushed a commit that referenced this issue Aug 25, 2023
@windtalker
Support Join auto spill (#7942)
3fec901 
ref #7738
@windtalker windtalker mentioned this issue Aug 28, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 29, 2023
@windtalker
Auto spill enhance (#8033)
aed7d84 
ref #7738
This was referenced Aug 29, 2023
Merged
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Aug 30, 2023
@windtalker
spill agg blocks using BlockInputStream (#8040)
15e13e0 
ref #7738
ti-chi-bot bot pushed a commit that referenced this issue Aug 30, 2023
@windtalker
support auto spill set by tidb (#8037)
794331c 
ref #7738
@windtalker windtalker mentioned this issue Aug 31, 2023
Closed
ti-chi-bot bot pushed a commit that referenced this issue Aug 31, 2023
@windtalker
Agg auto spill refine (#8038)
352100d 
ref #7738
@windtalker windtalker mentioned this issue Sep 1, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 4, 2023
@windtalker
Refine auto spill for fine grained executor (#8045)
569b567 
ref #7738
@windtalker windtalker mentioned this issue Sep 5, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 7, 2023
@windtalker
Refine API in IAggregateFunction/Aggregator to support aggregate on p…
c853837 
…art of the input `block` formally (#8056)

ref #7738
@windtalker windtalker mentioned this issue Sep 7, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 11, 2023
@windtalker
Fully support aggregate on part of a input block (#8064)
f1d2d46 
ref #7738
@windtalker windtalker mentioned this issue Sep 12, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 14, 2023
@windtalker
support agg resize callback (#8078)
f6e8ffa 
ref #7738
This was referenced Sep 15, 2023
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Sep 15, 2023
@windtalker
Fix MPPTask hang introduced by #8064 (#8101)
62f63e7 
ref #7738
ti-chi-bot bot pushed a commit that referenced this issue Sep 19, 2023
@windtalker
Add auto spill ut (#8096)
32974c1 
ref #7738
@windtalker windtalker mentioned this issue Sep 20, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 20, 2023
@windtalker
Spill in pipeline model not driven by input block (#8115)
45f9bb1 
ref #7738
@windtalker windtalker mentioned this issue Sep 22, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 22, 2023
@windtalker
add design doc for query level spill (#8132)
7c0c0f0 
ref #7738
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/feature-request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@windtalker