Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Subtask]: refactor blocking class operators to avoid OOM #13524

Open
1 of 5 tasks
Tracked by #10729
ouyuanning opened this issue Dec 15, 2023 · 3 comments
Open
1 of 5 tasks
Tracked by #10729

[Subtask]: refactor blocking class operators to avoid OOM #13524

ouyuanning opened this issue Dec 15, 2023 · 3 comments
Assignees
@m-schen
Labels
kind/subtask priority/p0 Critical feature that should be implemented in this version team/c2
Milestone
1.3.0

Comments

@ouyuanning
Copy link
Contributor

ouyuanning commented Dec 15, 2023 •
edited
Loading

Parent Issue

#10729

筛一遍所有的阻塞类算子，避免mpool申请过多内存的问题和内存占用过多问题
目前自己预计要处理的算子包含：
1、mergeorder
2、lock_op
3、on_duplicate_key

其他join相关的算子，由龙冉通过spill那个issue处理。这里处理非join相关的阻塞类算子

Detail of Subtask

This issue is a summary of related task issues:

Describe implementation you've considered

No response

Additional information

No response
@ouyuanning ouyuanning self-assigned this Dec 15, 2023
@ouyuanning ouyuanning mentioned this issue Dec 15, 2023
Open
15 tasks
@ouyuanning ouyuanning changed the title [Subtask]: refactor blocking class to use spills to avoid OOM [Subtask]: refactor blocking class operators to use spills to avoid OOM Dec 15, 2023
@ouyuanning ouyuanning added the priority/p0 Critical feature that should be implemented in this version label Dec 15, 2023
@ouyuanning ouyuanning added this to the 1.2.0 milestone Dec 15, 2023
@ouyuanning ouyuanning changed the title [Subtask]: refactor blocking class operators to use spills to avoid OOM [Subtask]: refactor blocking class operators to avoid OOM Dec 20, 2023
@ouyuanning
Copy link
Contributor Author

可以先筛选出要改造的算子清单。
然后挨个看使用什么方案处理，看是否有可以抽象的

@ouyuanning ouyuanning assigned m-schen and unassigned ouyuanning Dec 20, 2023
@ouyuanning ouyuanning mentioned this issue Dec 25, 2023
Closed
1 task
@florashi181
Copy link
Contributor

待进一步讨论和spill issue的分工和方案 #13640

@florashi181 florashi181 mentioned this issue Dec 28, 2023
Open
1 task
@florashi181
Copy link
Contributor

florashi181 commented Dec 28, 2023 •
edited
Loading

与内存issue 关联 #11509

@ouyuanning ouyuanning modified the milestones: 1.2.0, 1.3.0-Backlog Apr 1, 2024
@sukki37 sukki37 modified the milestones: 1.3.0-Backlog, 1.3.0 Jul 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@m-schen m-schen

Labels
kind/subtask priority/p0 Critical feature that should be implemented in this version team/c2
Projects
None yet
Milestone
1.3.0
Development

No branches or pull requests
4 participants
@ouyuanning @m-schen @sukki37 @florashi181