Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhance: calculate the accuracy memory usage while loading segment (#30473) #30475

Merged
merged 1 commit into from
Feb 3, 2024

Conversation

yah01
Copy link
Member

@yah01 yah01 commented Feb 2, 2024

the old version Knowhere would copy the index data while loading, we need to consider this to avoid OOM.

Knowhere provides a util function to indicate whether it will load the index with disk, if not, we need to double the memory usage prediction for index data

pr: #30473

@sre-ci-robot sre-ci-robot added the size/S Denotes a PR that changes 10-29 lines. label Feb 2, 2024
@mergify mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Feb 2, 2024
Copy link
Contributor

mergify bot commented Feb 2, 2024

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Copy link
Contributor

mergify bot commented Feb 2, 2024

@yah01 E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Copy link

codecov bot commented Feb 2, 2024

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (f50799b) 82.11% compared to head (a758da6) 82.16%.
Report is 1 commits behind head on 2.3.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##              2.3   #30475      +/-   ##
==========================================
+ Coverage   82.11%   82.16%   +0.04%     
==========================================
  Files         841      831      -10     
  Lines      121261   121192      -69     
==========================================
- Hits        99579    99572       -7     
+ Misses      18436    18385      -51     
+ Partials     3246     3235      -11     
Files Coverage Δ
internal/querynodev2/segments/segment_loader.go 77.19% <100.00%> (+0.35%) ⬆️
internal/core/src/segcore/load_index_c.cpp 36.58% <0.00%> (-0.70%) ⬇️

... and 164 files with indirect coverage changes

@mergify mergify bot added the ci-passed label Feb 2, 2024
the old version Knowhere would copy the index data while loading, we need to consider this to avoid OOM.

Knowhere provides a util function to indicate whether it will load the index with disk, if not, we need to double the memory usage prediction for index data

Signed-off-by: yah01 <yang.cen@zilliz.com>
@czs007
Copy link
Contributor

czs007 commented Feb 3, 2024

/approve
/lgtm

@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: czs007, yah01

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sre-ci-robot sre-ci-robot merged commit 655e235 into milvus-io:2.3 Feb 3, 2024
14 checks passed
sre-ci-robot pushed a commit that referenced this pull request Feb 6, 2024
…or (#30563)

pr: #30561

related pr: #30475

Signed-off-by: chyezh <chyezh@outlook.com>
sre-ci-robot pushed a commit that referenced this pull request Mar 1, 2024
…or (#30561)

related pr: #30475

Signed-off-by: chyezh <chyezh@outlook.com>
MrPresent-Han pushed a commit to MrPresent-Han/milvus that referenced this pull request Mar 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved ci-passed dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement lgtm size/S Denotes a PR that changes 10-29 lines.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants