-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Enhancement]: Pre-allocate space for datanode insert buffer preventing frequent grow slice #33561
Open
1 task done
Labels
Comments
congqixia
added a commit
to congqixia/milvus
that referenced
this issue
Jun 3, 2024
See also milvus-io#33561 This PR: - Adds a new param item for insert buffer chunk size - Pre-allocate for each insert buffer preventing frequent `growslice` Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
congqixia
added a commit
to congqixia/milvus
that referenced
this issue
Jun 11, 2024
See also milvus-io#33561 This PR: - Adds a new param item for insert buffer chunk size - Pre-allocate for each insert buffer preventing frequent `growslice` Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
sre-ci-robot
pushed a commit
that referenced
this issue
Jun 13, 2024
See also #33561 This PR: - Use zero copy when buffering insert messages - Make `storage.InsertCodec` support serialize multiple insert data chunk into same batch binlog files Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
congqixia
added a commit
to congqixia/milvus
that referenced
this issue
Jun 13, 2024
See also milvus-io#33561 milvus-io#33562 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
congqixia
added a commit
to congqixia/milvus
that referenced
this issue
Jun 13, 2024
See also milvus-io#33561 milvus-io#33562 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
sre-ci-robot
pushed a commit
that referenced
this issue
Jun 17, 2024
congqixia
added a commit
to congqixia/milvus
that referenced
this issue
Jun 26, 2024
…-io#33562) See also milvus-io#33561 This PR: - Use zero copy when buffering insert messages - Make `storage.InsertCodec` support serialize multiple insert data chunk into same batch binlog files Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
congqixia
added a commit
to congqixia/milvus
that referenced
this issue
Jun 26, 2024
…io#33817) See also milvus-io#33561 milvus-io#33562 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
yellow-shine
pushed a commit
to yellow-shine/milvus
that referenced
this issue
Jul 2, 2024
…-io#33562) See also milvus-io#33561 This PR: - Use zero copy when buffering insert messages - Make `storage.InsertCodec` support serialize multiple insert data chunk into same batch binlog files Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
yellow-shine
pushed a commit
to yellow-shine/milvus
that referenced
this issue
Jul 2, 2024
…io#33817) See also milvus-io#33561 milvus-io#33562 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Is there an existing issue for this?
What would you like to be added?
Pre-allocate insert buffer capacity in datanode
Why is this needed?
After profiling datanode CPU usage, the
growslice
happen frequently when write stress is heavyAnything else?
There is one drawback that the memory consumption may be notable when using partition key feature
The text was updated successfully, but these errors were encountered: