Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: DN OOM caused by Memory estimation quality of L0 comapction depending on the expansion rate of MemorySize of deltalog #34136

Closed
1 task done
XuanYang-cn opened this issue Jun 25, 2024 · 1 comment
Closed
1 task done

[Bug]: DN OOM caused by Memory estimation quality of L0 comapction depending on the expansion rate of MemorySize of deltalog #34136

XuanYang-cn opened this issue Jun 25, 2024 · 1 comment
Assignees
@XuanYang-cn
Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.

Comments

@XuanYang-cn
Copy link
Contributor

XuanYang-cn commented Jun 25, 2024 •
edited
Loading

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

  1. Add batchSize config to control the maximum batchSize
  2. Estimate the memoryUsage After loaded deltalogs of L0-Segment, Estimating shouldn't be trusting on MemorySize, it could be compressed size.
  • Use MemorySize of L0-segments to check if it'll OOM just to downloading those deltalogs.
  • After downloading deltalogs, use Actual memorySize of deltaLogs to estimate the affordable batchSize
  • If batchSize calculated is less than config.batchSize, use the smaller one.

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response
@XuanYang-cn XuanYang-cn added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jun 25, 2024
@XuanYang-cn XuanYang-cn self-assigned this Jun 25, 2024
@XuanYang-cn XuanYang-cn changed the title [Bug]: DN OOM caused by Memory estimation quality of L0 comapction depends on the expansion rate of MemorySize of deltalog [Bug]: DN OOM caused by Memory estimation quality of L0 comapction depending on the expansion rate of MemorySize of deltalog Jun 25, 2024
XuanYang-cn added a commit to XuanYang-cn/milvus that referenced this issue Jun 26, 2024
@XuanYang-cn
fix: DataNode might OOM by estimating based on MemorySize
9ac33b7 
See also: milvus-io#34136

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
@XuanYang-cn XuanYang-cn mentioned this issue Jun 26, 2024
Merged
XuanYang-cn added a commit to XuanYang-cn/milvus that referenced this issue Jun 26, 2024
@XuanYang-cn
fix: DataNode might OOM by estimating based on MemorySize
4d5f1e1 
See also: milvus-io#34136
pr: milvus-io#34201

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
@XuanYang-cn XuanYang-cn mentioned this issue Jun 26, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue Jul 4, 2024
@XuanYang-cn
fix: DataNode might OOM by estimating based on MemorySize (#34201)
8a2be8a 
See also: #34136

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Jul 4, 2024
@XuanYang-cn
fix: DataNode might OOM by estimating based on MemorySize (#34203)
0f1915e 
See also: #34136
pr: #34201

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
@XuanYang-cn
Copy link
Contributor Author

Should be fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@XuanYang-cn XuanYang-cn

Labels
kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@XuanYang-cn