enhance: Avoid merging insert data when buffering insert msgs #33562
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: congqixia The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
mergify
bot
added
dco-passed
|
@congqixia E2e jenkins job failed, comment |
1 similar comment
|
@congqixia E2e jenkins job failed, comment |
|
/run-cpu-e2e |
|
@congqixia E2e jenkins job failed, comment |
xiaofan-luan
reviewed
Jun 3, 2024
| return ib.currentBuffer().buffer | ||
| } | ||
| // no error assumed, buffer created before | ||
| result, _ := storage.NewInsertDataWithCap(ib.collSchema, int(ib.rows)) |
There was a problem hiding this comment.
maybe we don't really need to return a real *storage.InsertData ?
can we instead return a insertDataInterator? so we can avoid one extra copy?
There was a problem hiding this comment.
Good point. Working on it
There was a problem hiding this comment.
this shall be a major refactory in storage. we shall implement iterator in next PR
See also milvus-io#33561 This PR: - Adds a new param item for insert buffer chunk size - Pre-allocate for each insert buffer preventing frequent `growslice` Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
congqixia
force-pushed
the
enhance/preallocate_insert_buffer
branch
from
b48248e
to
ca7bae6
Compare
|
@congqixia E2e jenkins job failed, comment |
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #33562 +/- ##
==========================================
+ Coverage 80.88% 80.90% +0.01%
==========================================
Files 1051 1058 +7
Lines 134863 135201 +338
==========================================
+ Hits 109088 109387 +299
- Misses 21590 21621 +31
- Partials 4185 4193 +8
|
tedxu
reviewed
Jun 11, 2024
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
congqixia
changed the title
|
@congqixia ut workflow job failed, comment |
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
sre-ci-robot
added
size/XL
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
|
/lgtm |
yanliang567
added
ci-passed
manual-pass
congqixia
added a commit
to congqixia/milvus
that referenced
this pull request
Jun 13, 2024
See also milvus-io#33561 milvus-io#33562 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
congqixia
added a commit
to congqixia/milvus
that referenced
this pull request
Jun 13, 2024
See also milvus-io#33561 milvus-io#33562 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
sre-ci-robot
pushed a commit
that referenced
this pull request
Jun 17, 2024
congqixia
added a commit
to congqixia/milvus
that referenced
this pull request
Jun 26, 2024
…-io#33562) See also milvus-io#33561 This PR: - Use zero copy when buffering insert messages - Make `storage.InsertCodec` support serialize multiple insert data chunk into same batch binlog files Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
congqixia
added a commit
to congqixia/milvus
that referenced
this pull request
Jun 26, 2024
…io#33817) See also milvus-io#33561 milvus-io#33562 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
yellow-shine
pushed a commit
to yellow-shine/milvus
that referenced
this pull request
Jul 2, 2024
…-io#33562) See also milvus-io#33561 This PR: - Use zero copy when buffering insert messages - Make `storage.InsertCodec` support serialize multiple insert data chunk into same batch binlog files Signed-off-by: Congqi Xia <congqi.xia@zilliz.com> --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
yellow-shine
pushed a commit
to yellow-shine/milvus
that referenced
this pull request
Jul 2, 2024
…io#33817) See also milvus-io#33561 milvus-io#33562 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See also #33561
This PR:
storage.InsertCodecsupport serialize multiple insert data chunk into same batch binlog filesSigned-off-by: Congqi Xia congqi.xia@zilliz.com