Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature]: Bulk load data #15604

Closed
1 task done
yhmo opened this issue Feb 17, 2022 · 6 comments
Closed
1 task done

[Feature]: Bulk load data #15604

yhmo opened this issue Feb 17, 2022 · 6 comments
Labels
kind/feature Issues related to feature request from users stale indicates no udpates for 30 days

Comments

@yhmo
Copy link
Contributor

yhmo commented Feb 17, 2022 •
edited

Is there an existing issue for this?

  • I have searched the existing issues

Is your feature request related to a problem? Please describe.

Typically, it cost several (5~6) hours to insert one billion entities with 128-dimensional vectors. We need a new interface to do bulk load for the following purposes:

  1. import data from JSON format files. (first stage)
  2. import data from Numpy format files. (first stage)
  3. copy a collection within on Milvus 2.0 service. (second stage)
  4. copy a collection from one Milvus 2.0 service to another. (second stage)
  5. import data from Milvus 1.x to Milvus 2.0 (third stage)
  6. parquet/faiss files (TBD)

Describe the solution you'd like.

No response

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response
@yhmo yhmo added the kind/feature Issues related to feature request from users label Feb 17, 2022
@xiaofan-luan
Copy link
Contributor

what kind of format are we plan to support? Json, parquet, xml or ...

@yhmo
Copy link
Contributor Author

yhmo commented Feb 17, 2022

what kind of format are we plan to support? Json, parquet, xml or ...

Basically json/numpy.

@xiaofan-luan
Copy link
Contributor

what kind of format are we plan to support? Json, parquet, xml or ...

Basically json/numpy.

should be enough as the first release, later on we can support parquet, faiss and even more.

This was referenced Mar 8, 2022
Merged
Merged
@stale
Copy link

stale bot commented Mar 20, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label Mar 20, 2022
This was referenced Mar 21, 2022
Merged
Merged
Merged
@soothing-rain soothing-rain mentioned this issue Mar 23, 2022
Merged
soothing-rain added a commit to soothing-rain/milvus that referenced this issue Mar 25, 2022
@soothing-rain
Add Etcd support for Import Tasks.
5b0440d 
issue: milvus-io#15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Mar 25, 2022
@soothing-rain
Add Etcd support for Import Tasks. (#16164)
a23ca74 
issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
soothing-rain added a commit to soothing-rain/milvus that referenced this issue Mar 25, 2022
@soothing-rain
Support check index state in root coord, for bulk load feature.
9c090ee 
issue: milvus-io#15604

/kind feature

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
@soothing-rain soothing-rain mentioned this issue Mar 25, 2022
Merged
soothing-rain added a commit to soothing-rain/milvus that referenced this issue Mar 25, 2022
@soothing-rain
Support check index state in root coord, for bulk load feature.
13c173a 
issue: milvus-io#15604

/kind feature

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
@stale stale bot closed this as completed Mar 27, 2022
soothing-rain added a commit to soothing-rain/milvus that referenced this issue Mar 28, 2022
@soothing-rain
Implement ReportImport related logic.
e566232 
/kind feature

issue: milvus-io#15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
soothing-rain added a commit to soothing-rain/milvus that referenced this issue Mar 28, 2022
@soothing-rain
Support check index state in root coord, for bulk load feature.
d472a6f 
issue: milvus-io#15604

/kind feature

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Mar 28, 2022
@soothing-rain
Support check index state in root coord, for bulk load feature. (#16198)
40c703d 
issue: #15604

/kind feature

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
@soothing-rain soothing-rain mentioned this issue Mar 28, 2022
Closed
soothing-rain added a commit to soothing-rain/milvus that referenced this issue Apr 20, 2022
@soothing-rain
Make bulk load fully work
3d3444b 
issue: milvus-io#15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Apr 20, 2022
@soothing-rain
Make bulk load fully work (#16512)
289e468 
issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
This was referenced Apr 20, 2022
Merged
Merged
Merged
@soothing-rain soothing-rain mentioned this issue Apr 24, 2022
Merged
soothing-rain added a commit to soothing-rain/milvus that referenced this issue Apr 24, 2022
@soothing-rain
Resolve an issue with bulk load where segments in flow graph replica …
e0380cb 
…are not updated

issue: milvus-io#15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
soothing-rain added a commit to soothing-rain/milvus that referenced this issue Apr 24, 2022
@soothing-rain
Fix an issue where query coord calls MinioChunkManager during init
bf771b9 
It causes Milvus not runnable with local storage.

issue: milvus-io#15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
soothing-rain added a commit to soothing-rain/milvus that referenced this issue Apr 24, 2022
@soothing-rain
Fix an issue where query coord calls MinioChunkManager during init
17c669f 
It causes Milvus not runnable with local storage.

issue: milvus-io#15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
@soothing-rain soothing-rain mentioned this issue Apr 24, 2022
Merged
sre-ci-robot pushed a commit that referenced this issue Apr 25, 2022
@soothing-rain
Resolve an issue with bulk load where segments in flow graph replica …
e66ac6a 
…are not updated (#16609)

issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
sre-ci-robot pushed a commit that referenced this issue Apr 25, 2022
@soothing-rain
Fix an issue where query coord calls MinioChunkManager during init (#…
4ef2df8 
…16629)

It causes Milvus not runnable with local storage.

issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
@yhmo yhmo mentioned this issue Apr 27, 2022
Merged
soothing-rain added a commit to soothing-rain/milvus that referenced this issue May 12, 2022
@soothing-rain
Fix an issue where query coord calls MinioChunkManager during init (m…
4b4efbe 
…ilvus-io#16629)

It causes Milvus not runnable with local storage.

issue: milvus-io#15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
@stale
Copy link

stale bot commented May 18, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

@stale stale bot added the stale indicates no udpates for 30 days label May 18, 2022
@stale stale bot closed this as completed May 25, 2022
This was referenced Jun 17, 2022
Merged
Closed
@yhmo yhmo mentioned this issue Jun 30, 2022
Closed
This was referenced Sep 26, 2022
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/feature Issues related to feature request from users stale indicates no udpates for 30 days
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@yhmo @xiaofan-luan @claireyuw