Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Load may become slower with large dataset with prewarm #32772

Closed
1 task done
xiaofan-luan opened this issue May 6, 2024 · 2 comments
Closed
1 task done

[Bug]: Load may become slower with large dataset with prewarm #32772

xiaofan-luan opened this issue May 6, 2024 · 2 comments
Assignees
@xiaofan-luan
Labels
kind/bug Issues or changes related a bug stale indicates no udpates for 30 days triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@xiaofan-luan
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

  1. 2.3.6 introduced warm up.
  2. the warm up use load pool, and there left two issues:
    1. warm up may block further load
    2. load pool is blocking, so if the pool thread is full the load will also be blocked

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response
@xiaofan-luan xiaofan-luan added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 6, 2024
@xiaofan-luan xiaofan-luan mentioned this issue May 6, 2024
Closed
@xiaofan-luan xiaofan-luan changed the title [Bug]: Load may become slower with large dataset [Bug]: Load may become slower with large dataset with prewarm May 6, 2024
@yanliang567
Copy link
Contributor

/assign @xiaofan-luan
/unassign

@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 7, 2024
This was referenced May 24, 2024
Merged
Merged
sre-ci-robot pushed a commit that referenced this issue May 26, 2024
@bigsheeper @xiaofan-luan
fix: use seperate warmup pool and disable warmup by default (#33348)
760223f 
1. use a small warmup pool to reduce the impact of warmup
2. change the warmup pool to nonblocking mode
3. disable warmup by default
4. remove the maximum size limit of 16 for the load pool

issue: #32772

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: xiaofanluan <xiaofan.luan@zilliz.com>
@bigsheeper bigsheeper mentioned this issue May 28, 2024
Merged
sre-ci-robot pushed a commit that referenced this issue May 28, 2024
@bigsheeper @xiaofan-luan
fix: use seperate warmup pool and disable warmup by default (#33348) (#…
7384bfe 
…33349)

1. use a small warmup pool to reduce the impact of warmup
2. change the warmup pool to nonblocking mode
3. disable warmup by default
4. remove the maximum size limit of 16 for the load pool

issue: #32772

pr: #33348

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: xiaofanluan <xiaofan.luan@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue May 29, 2024
@bigsheeper
enhance: Replace 'off' with 'disable' (#33433)
bbb6998 
YAML will automatically parse "off" as a boolean variable. We should
avoid using "off" in the future.

issue: #32772

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
@stale Stale
Copy link

stale bot commented Jun 6, 2024

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Jun 6, 2024
@stale stale bot closed this as completed Jun 16, 2024
yellow-shine pushed a commit to yellow-shine/milvus that referenced this issue Jul 2, 2024
@bigsheeper @xiaofan-luan @yellow-shine
fix: use seperate warmup pool and disable warmup by default (milvus-i…
e4cfe6a 
…o#33348)

1. use a small warmup pool to reduce the impact of warmup
2. change the warmup pool to nonblocking mode
3. disable warmup by default
4. remove the maximum size limit of 16 for the load pool

issue: milvus-io#32772

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: xiaofanluan <xiaofan.luan@zilliz.com>
yellow-shine pushed a commit to yellow-shine/milvus that referenced this issue Jul 2, 2024
@bigsheeper @yellow-shine
enhance: Replace 'off' with 'disable' (milvus-io#33433)
f6ca3ef 
YAML will automatically parse "off" as a boolean variable. We should
avoid using "off" in the future.

issue: milvus-io#32772

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xiaofan-luan xiaofan-luan

Labels
kind/bug Issues or changes related a bug stale indicates no udpates for 30 days triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: use seperate warmup pool and make it non blocking xiaofan-luan/milvus
2 participants
@yanliang567 @xiaofan-luan