-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: querynode cpu larger and larger #34270
Comments
generally speaking, insertion does not matter to the cpu usage of quernode, it could impact the cpu usage of datanode. so quick questions:
|
1.insert batch , batch is 119 ,emb size is 128. and the concurrencies may be 1500 |
|
but the cpu is larger and larger ,even up to 100%. and many search is timeout. I try to release the collection and rebuild the index. and then the search is normal, the cpu is not high. |
you might need to increase index node number if there are many segment lefted unindexed |
or use ivf since it's build speed is much faster compared to hnsw |
I think the indexnoed is enough , original the indexnode num is 5. and sometime the cpu is 0. the indexnode is 16 core 16g. I try use 12 indexnodes . but the querynode cpu is also high. it might not nothing to do with indexnode. |
I use IVF_SQ8 index type |
what about the memory usage of quernodes at that moment? We still need full milvus logs to know what was happening, or we could just guess everything...:( |
whether have other method to export the log, I can enter every node to run some command. |
the print log is: |
datanode: |
@lzhin I saw error logs Policies not found for public/default namespace and Failed to connect to broker. |
I dont know . this is the pulsar log: |
Is there an existing issue for this?
Environment
Current Behavior
as the emb insert to collection , the querynode cpu larger and larger. it seems some index not build in time
![image](https://private-user-images.githubusercontent.com/17926066/344141586-9b76f25f-c1d8-4830-a0ba-15dd6f7c566d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA5MzYzNDQsIm5iZiI6MTcyMDkzNjA0NCwicGF0aCI6Ii8xNzkyNjA2Ni8zNDQxNDE1ODYtOWI3NmYyNWYtYzFkOC00ODMwLWEwYmEtMTVkZDZmN2M1NjZkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA3MTQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNzE0VDA1NDcyNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTkyNTQyMWExNGFkZmE4NTRmNjY2OTQxOTAyODViZGQyN2Q3MzUzMjA3OTU0NzI0MGU3YjliNDBkNmE2MmQxM2QmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.sQs1Eep4jh1MkByhG51pq4ikQ6RfTF_vUl-nAU150SE)
Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
No response
Anything else?
No response
The text was updated successfully, but these errors were encountered: