Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] Pure CPU version for Milvus #12

Closed
yhmo opened this issue Oct 16, 2019 · 3 comments
Closed

[FEATURE] Pure CPU version for Milvus #12

yhmo opened this issue Oct 16, 2019 · 3 comments
Assignees
@youny626
Labels
kind/enhancement Issues or changes related to enhancement

Comments

@yhmo
Copy link
Contributor

yhmo commented Oct 16, 2019

Is your feature request related to a problem? Please describe.
Some developers don't have Nvidia gpu card on their machine, they cannot compile/run Milvus.

Describe the solution you'd like
Use pre-compiled macro to disable gpu function code
@yhmo yhmo added the kind/enhancement Issues or changes related to enhancement label Oct 24, 2019
@youny626 youny626 changed the title [FEATURE]Pure cpu version for Milvus [FEATURE]Pure CPU version for Milvus Oct 25, 2019
@youny626 youny626 changed the title [FEATURE]Pure CPU version for Milvus [FEATURE] Pure CPU version for Milvus Oct 28, 2019
This was referenced Nov 1, 2019
Closed
Merged
@cxie cxie mentioned this issue Nov 5, 2019
Closed
@da2r-20
Copy link

da2r-20 commented Nov 7, 2019

Can we have a CPU only docker?

@yhmo
Copy link
Contributor Author

yhmo commented Nov 7, 2019

Sure, we will provide cpu and gpu docker image separately when we release 0.6.0.

@youny626 youny626 mentioned this issue Nov 7, 2019
Merged
@cydrain
Copy link
Contributor

cydrain commented Dec 11, 2019

fixed

@cydrain cydrain closed this as completed Dec 11, 2019
@cydrain cydrain mentioned this issue Dec 14, 2019
Closed
@lale314 lale314 mentioned this issue Jan 13, 2020
Closed
yhmo pushed a commit that referenced this issue Jan 17, 2020
@shengjun1985
Merge pull request #12 from milvus-io/crud
c409aaf 
Crud
@op-hunter op-hunter mentioned this issue Apr 11, 2020
Closed
@jhd900518 jhd900518 mentioned this issue Dec 8, 2020
Closed
@xuqinkun xuqinkun mentioned this issue Jul 21, 2021
Closed
@bigsheeper bigsheeper mentioned this issue Nov 29, 2021
Closed
1 task
@qi49125 qi49125 mentioned this issue Feb 18, 2022
Closed
1 task
@lijonnas lijonnas mentioned this issue Mar 26, 2022
Closed
1 task
This was referenced Jun 6, 2022
Closed
Closed
wayblink referenced this issue in wayblink/milvus Jul 19, 2022
@wayblink
Persist import task info to meta before update in memory (#12)
72e97c1 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
wayblink referenced this issue in wayblink/milvus Aug 12, 2022
@wayblink @soothing-rain
Persist import task info to meta before update in memory (#12)
98d6692 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
yhmo pushed a commit to yhmo/milvus that referenced this issue Aug 26, 2022
@wayblink @soothing-rain
Persist import task info to meta before update in memory (milvus-io#12)
5892434 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
yhmo added a commit to yhmo/milvus that referenced this issue Sep 16, 2022
@soothing-rain @EricStarer
@wayblink @yhmo
Squash (milvus-io#25)
cd77165 
* Refine add segment logic for bulk load

Also restructured `addNewSegment`, `addNormalSegment` and `addFlushedSegment` into a single `addSegment` to avoid duplicate code block and to see more clearly how adding these segments are different.

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Add garbage collection function for segments produced by partially failed imports (#6)

Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>

* Enhance error handle in bulk load (#5)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Refine expiration/retention policy for import tasks (milvus-io#9)

* Persist import task info to meta before update in memory (milvus-io#12)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Add start position for import segment (milvus-io#7)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Refine bulkload proto (milvus-io#11)

Signed-off-by: groot <yihua.mo@zilliz.com>

* Remove queryable/indexed logic and some other refinement (milvus-io#14)

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Resolve conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Some minor tweaks

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Implement CompleteBulkLoad feature (milvus-io#19)

* Update interface and fix bug

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix dml start position mistake in AddSegmentReq (milvus-io#21)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Fix getDmlChannelPositionByBroadcast (milvus-io#23)

* Fix null start position issue

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix merge conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Bulkload enhancement (milvus-io#24)

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* Add some unit tests

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: xing.zhao <34002927+EricStarer@users.noreply.github.com>
Co-authored-by: xingzhao <xing.zhao@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: groot <yihua.mo@zilliz.com>
yhmo added a commit to yhmo/milvus that referenced this issue Sep 20, 2022
@soothing-rain @EricStarer
@wayblink @yhmo
Squash (milvus-io#25)
6745eb4 
* Refine add segment logic for bulk load

Also restructured `addNewSegment`, `addNormalSegment` and `addFlushedSegment` into a single `addSegment` to avoid duplicate code block and to see more clearly how adding these segments are different.

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Add garbage collection function for segments produced by partially failed imports (#6)

Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>

* Enhance error handle in bulk load (#5)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Refine expiration/retention policy for import tasks (milvus-io#9)

* Persist import task info to meta before update in memory (milvus-io#12)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Add start position for import segment (milvus-io#7)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Refine bulkload proto (milvus-io#11)

Signed-off-by: groot <yihua.mo@zilliz.com>

* Remove queryable/indexed logic and some other refinement (milvus-io#14)

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Resolve conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Some minor tweaks

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Implement CompleteBulkLoad feature (milvus-io#19)

* Update interface and fix bug

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix dml start position mistake in AddSegmentReq (milvus-io#21)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Fix getDmlChannelPositionByBroadcast (milvus-io#23)

* Fix null start position issue

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix merge conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Bulkload enhancement (milvus-io#24)

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* Add some unit tests

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: xing.zhao <34002927+EricStarer@users.noreply.github.com>
Co-authored-by: xingzhao <xing.zhao@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: groot <yihua.mo@zilliz.com>
wayblink referenced this issue in wayblink/milvus Sep 22, 2022
@soothing-rain @EricStarer
@wayblink @yhmo
Squash (#25)
bb92d0a 
* Refine add segment logic for bulk load

Also restructured `addNewSegment`, `addNormalSegment` and `addFlushedSegment` into a single `addSegment` to avoid duplicate code block and to see more clearly how adding these segments are different.

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Add garbage collection function for segments produced by partially failed imports (#6)

Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>

* Enhance error handle in bulk load (#5)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Refine expiration/retention policy for import tasks (#9)

* Persist import task info to meta before update in memory (#12)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Add start position for import segment (#7)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Refine bulkload proto (#11)

Signed-off-by: groot <yihua.mo@zilliz.com>

* Remove queryable/indexed logic and some other refinement (#14)

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Resolve conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Some minor tweaks

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Implement CompleteBulkLoad feature (#19)

* Update interface and fix bug

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix dml start position mistake in AddSegmentReq (#21)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Fix getDmlChannelPositionByBroadcast (#23)

* Fix null start position issue

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix merge conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Bulkload enhancement (#24)

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* Add some unit tests

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: xing.zhao <34002927+EricStarer@users.noreply.github.com>
Co-authored-by: xingzhao <xing.zhao@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: groot <yihua.mo@zilliz.com>
yhmo added a commit to yhmo/milvus that referenced this issue Sep 23, 2022
@soothing-rain @EricStarer
@wayblink @yhmo
Squash (milvus-io#25)
ca56af5 
* Refine add segment logic for bulk load

Also restructured `addNewSegment`, `addNormalSegment` and `addFlushedSegment` into a single `addSegment` to avoid duplicate code block and to see more clearly how adding these segments are different.

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Add garbage collection function for segments produced by partially failed imports (#6)

Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>

* Enhance error handle in bulk load (#5)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Refine expiration/retention policy for import tasks (milvus-io#9)

* Persist import task info to meta before update in memory (milvus-io#12)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Add start position for import segment (milvus-io#7)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Refine bulkload proto (milvus-io#11)

Signed-off-by: groot <yihua.mo@zilliz.com>

* Remove queryable/indexed logic and some other refinement (milvus-io#14)

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Resolve conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Some minor tweaks

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Implement CompleteBulkLoad feature (milvus-io#19)

* Update interface and fix bug

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix dml start position mistake in AddSegmentReq (milvus-io#21)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Fix getDmlChannelPositionByBroadcast (milvus-io#23)

* Fix null start position issue

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix merge conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Bulkload enhancement (milvus-io#24)

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* Add some unit tests

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: xing.zhao <34002927+EricStarer@users.noreply.github.com>
Co-authored-by: xingzhao <xing.zhao@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: groot <yihua.mo@zilliz.com>
yhmo added a commit to yhmo/milvus that referenced this issue Sep 24, 2022
@soothing-rain @EricStarer
@wayblink @yhmo
Squash (milvus-io#25)
678ec27 
* Refine add segment logic for bulk load

Also restructured `addNewSegment`, `addNormalSegment` and `addFlushedSegment` into a single `addSegment` to avoid duplicate code block and to see more clearly how adding these segments are different.

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Add garbage collection function for segments produced by partially failed imports (#6)

Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>

* Enhance error handle in bulk load (#5)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Refine expiration/retention policy for import tasks (milvus-io#9)

* Persist import task info to meta before update in memory (milvus-io#12)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Add start position for import segment (milvus-io#7)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Refine bulkload proto (milvus-io#11)

Signed-off-by: groot <yihua.mo@zilliz.com>

* Remove queryable/indexed logic and some other refinement (milvus-io#14)

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Resolve conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Some minor tweaks

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Implement CompleteBulkLoad feature (milvus-io#19)

* Update interface and fix bug

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix dml start position mistake in AddSegmentReq (milvus-io#21)

Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>

* Fix getDmlChannelPositionByBroadcast (milvus-io#23)

* Fix null start position issue

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix merge conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Fix conflicts

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

* Bulkload enhancement (milvus-io#24)

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* Add some unit tests

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: groot <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: xing.zhao <34002927+EricStarer@users.noreply.github.com>
Co-authored-by: xingzhao <xing.zhao@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: groot <yihua.mo@zilliz.com>
@NicoYuan1986 NicoYuan1986 mentioned this issue Jul 28, 2023
Closed
1 task
@yiwangdr yiwangdr mentioned this issue Sep 15, 2023
Closed
1 task
@zhaowenZhou zhaowenZhou mentioned this issue Nov 23, 2023
Closed
1 task
@zhagnlu zhagnlu mentioned this issue Jul 23, 2024
Closed
1 task
Suvhrajit pushed a commit to Suvhrajit/milvus that referenced this issue Jul 27, 2024
@chern
@W-16154282 Initialize S3 client with session token conditionally, cr…
c9ffa70 
…eate BYOK enabled flag (milvus-io#12)

Conditioned on feature flag

---------

Co-authored-by: Sriram Ganesh <sriram.ganesh@salesforce.com>
Suvhrajit pushed a commit to Suvhrajit/milvus that referenced this issue Jul 27, 2024
@chern
@W-16154282 Initialize S3 client with session token conditionally, cr…
412cdb1 
…eate BYOK enabled flag (milvus-io#12)

Conditioned on feature flag

---------

Co-authored-by: Sriram Ganesh <sriram.ganesh@salesforce.com>
Signed-off-by: Suvhrajit <suvhrajit.basak@salesforce.com>
Suvhrajit pushed a commit to Suvhrajit/milvus that referenced this issue Aug 8, 2024
@chern
@W-15991930 Add codeowners (milvus-io#2)
7af4a51 
@W-16056539 Remove certs leftover from open-source repo (milvus-io#4)

Setup proto and grpc client for access manger

Setup gRPC access manager client

@W-16002482 Add session token and KMS key fields to StorageConfig (milvus-io#10)

@W-16154282 Initialize S3 client with session token conditionally, create BYOK enabled flag (milvus-io#12)

Conditioned on feature flag

---------

Co-authored-by: Sriram Ganesh <sriram.ganesh@salesforce.com>

@W-16315979 Use SSE KMS Key for S3 PutObject if kms_key_id is present (milvus-io#18)

Setup proto and grpc client for access manger

Setup gRPC access manager client

@W-16256135 Getting credentials from dpc-cvs-access-manager - Part 1

@W-16161994 - Instantiate Remote Chunk Managers per collectionId

@W-16256135 Getting credentials from dpc-cvs-access-manager - Part 1

refactor'd name

added to cmake

created time conversion helper method

revert to 2.3 conanfile

added whitespace back

fixed make

addressed comment

@W-16256135 Making use of collectionChunkManagerCache

@W-16256135 Adding TODO in index_c.cpp

@W-16256135 Addressing comments

@W-16256135 Addressing more comments

@W-16256135 Addressing more comments - 2

Trying .strata and sonarqube

Addressing comments

byok flag fix

collection manager initialization

@W-16368067 Fix chunk manager (milvus-io#36)

This PR is to fix chunk manager changes. We now create chunk manager
that creates an access manager client to get credentials for the tenant.
It was previously not working. This PR has multiple bug fixes to get
chunk manager going.

@W-15929693 - Milvus 2.3 Index file changes , rc - incorporated

extended chunk manager to manage collection based chunk manager

access manager go integration

all the changes

e2e changes fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@youny626 youny626

Labels
kind/enhancement Issues or changes related to enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@yhmo @cydrain @da2r-20 @youny626