Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The index size is wrong #2890

Closed
shengjun1985 opened this issue Jul 16, 2020 · 0 comments · Fixed by #2930 or #2942
Closed

The index size is wrong #2890

shengjun1985 opened this issue Jul 16, 2020 · 0 comments · Fixed by #2930 or #2942
Assignees
Labels
kind/bug Issues or changes related a bug

Comments

@shengjun1985
Copy link
Contributor

shengjun1985 commented Jul 16, 2020

Please state you issue using the following template and, most importantly, in English.

Describe the bug
Now, the system is using index file size to indicate its size in memory.
But sometimes it is a terrible mistake. Please refer to issue #2803
Thus, it will cause the cache policy not to work well.

Steps/Code to reproduce behavior
none

Expected behavior
Correct the index size.

Environment details

  • Milvus version (v0.10.0)

Screenshots
none

Additional context
Here is the wrong logic.
DefaultVectorIndexFormat::read_internal()
index->SetIndexSize(length);

@shengjun1985 shengjun1985 added the kind/bug Issues or changes related a bug label Jul 16, 2020
@shengjun1985 shengjun1985 self-assigned this Jul 16, 2020
This was referenced Jul 17, 2020
@mergify mergify bot closed this as completed in #2930 Jul 21, 2020
mergify bot pushed a commit that referenced this issue Jul 21, 2020
* fix update index size

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* add GetSize() interface 4 SPTAG

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix index size of index hnsw, annoy and nsg

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix clang

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* add sdk 4 non-ivf index test

Signed-off-by: cmli <chengming.li@zilliz.com>

* remove std::cout and make clang-format

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix cout

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* checkout sdk

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix binary ivf

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* fix hnsw

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

Co-authored-by: cmli <chengming.li@zilliz.com>
Co-authored-by: shengjun.li <shengjun.li@zilliz.com>
mergify bot pushed a commit that referenced this issue Jul 21, 2020
* fix NM index size

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* add raw data size of hnsw_nm and nsg_nm

Signed-off-by: cmli <chengming.li@zilliz.com>

Co-authored-by: cmli <chengming.li@zilliz.com>
fishpenguin added a commit that referenced this issue Jul 25, 2020
* fix a typo (#2712)

Signed-off-by: cmli <chengming.li@zilliz.com>

Co-authored-by: cmli <chengming.li@zilliz.com>

* snapshot integration (#2716)

* code opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add some APIs for SSDBImpl

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* Web fix 2690 (#2691)

* Remove body parser in show partitions endpoints (fix #2690)

Signed-off-by: yhz <413554850@qq.com>

* Remove surplus output and comments

Signed-off-by: yhz <413554850@qq.com>

* Remove surplus output and comments

Signed-off-by: yhz <413554850@qq.com>

* Remove space line

Signed-off-by: yhz <413554850@qq.com>

* Remove surplus ut

Signed-off-by: yhz <413554850@qq.com>

* laten sleep time in case test_delete_vector_collection_count_no_flush

Signed-off-by: yhz <413554850@qq.com>

* #2689 Construct Knowhere Index Without Data (#2704)

* Offset for IVFFlat

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* modify segment reader

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* add index_flat_nm

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* fix compilation issue

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* fix segment reader

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* Fix issue

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* fix hnsw_nm

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* fix nsg

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* fix bug

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* NSG

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* Fix NSG issue

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* client test

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* fix index size

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* remove unnecessary things

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* add changelog

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* Fix clang format

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* add changelog

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* Fix compile error

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* Fix compile error

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* Fix compile error

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* Fix compile error

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* Fix issues

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* Change data to shared_ptr

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* fix hnsw and nsg ut

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* clang-format

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* not to insert cache immediately

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* Fix macro issue

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

Co-authored-by: shengjun.li <shengjun.li@zilliz.com>

* (db/snapshot): Code Integration (#2718)

* (db/snapshot): integrate stage 1

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): integrate stage 2

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): integrate stage 3

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix ut

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): integrate stage 4

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): integrate stage 5

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): integrate stage 6

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): integrate stage 7

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add params for some resources

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add field type

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): implement iterate framework for Snapshot

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): refactor code

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint errors

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add SnapshotHandlers

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add SnapshotVistor

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add SizeField for all resource commit

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add SegmentsToSearch collector

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Iterate handler update

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Update IterateHandler

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add some hooks for IterateHandler

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Small code refactor

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add const for some snapshot APIs

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add const for some snapshots APIs

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Update create collection

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): merge to new ssdb

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint errors

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add test_db in ssdb

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update test_ssdb

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add PartitionTest

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add DropIndexOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add IndexTest

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Fix some bugs

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add RowCount for some resources

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): code refactor

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix compile error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* Add pipeline step to print/get the container logs (#2730)

* Add pipeline step to print/get the container logs

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* Add pipeline step to print/get the container logs

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* Create index failed caused by server crashed (#2736)

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

Co-authored-by: groot <yhmo@zeronedata.com>

* (db/snapshot): Add DropAllIndexOperation and related UT (#2737)

* (db/snapshot): add drop all index operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add ut for drop all index operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* search combine configable

Signed-off-by: groot <yihua.mo@zilliz.com>

* typo

Signed-off-by: groot <yihua.mo@zilliz.com>

* Increase GRPC concurrency (#2671)

* Increase GRPC concurrency

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* Fix lint

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* Add limitation

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* Fix compilation error

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* Fix compilation error

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* Fix compilation error

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* Change Milvus Helm Version to v0.10.1 (#2678)

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* [skip ci] update nightly cron  (#2680)

* Change Milvus Helm Version to v0.10.1

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* [skip ci] update nightly cron

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* Update nightly cron (#2681)

* Change Milvus Helm Version to v0.10.1

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* [skip ci] update nightly cron

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* [skip ci] update nightly cron

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* * #2642 Create index failed caused by server crashed.

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* changelog

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* changelog

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* enable ivf_pq <ip> search cases on GPU (#2708)

* update cron on 0.10.1

Signed-off-by: zw <zw@milvus.io>

* enable ivf_pq <ip> search cases on GPU

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* * #2642 Create index failed caused by server crashed.

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* changelog

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* changelog

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* #2692 Milvus hangs during multi-thread concurrent search

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* Fix compiling error

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* [skip ci] Update PR template

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* Let codec singleton (#2742)

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* (db/snapshot): Bug fix in operations and unittest (#2744)

* (db/snapshot): fix bugs and update UT

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update Snapshot ToString

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add DropIndex API

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add DropIndex API UT

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add more APIs for Snapshot (#2747)

* (db/snapshot): Add more APIs for Snapshot

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Refactor field element related code

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove some prints

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* add background threads to SSDBImpl (#2751)

* add background threads to SSDBImpl

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* code format

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* snapshot integrate GetEntityByID (#2753)

* code opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add some APIs for SSDBImpl

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* partially add GetVectorById

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* snapshot opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix typo

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update GetVectorByID framework

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename GetResFiles to GetResPath

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update GetVectorByIdSegmentHandler

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add GetEntityByID

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update DataType

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update ParamField

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix: upgrade com.alibaba:fastjson from 1.2.70 to 1.2.71 (#2756)

Snyk has created this PR to upgrade com.alibaba:fastjson from 1.2.70 to 1.2.71.

See this package in NPM:
https://www.npmjs.com/package/com.alibaba:fastjson

See this project in Snyk:
https://app.snyk.io/org/cxie/project/2f701ce6-b421-4bbf-aaad-9d2bdb8569e4?utm_source=github&utm_medium=upgrade-pr

* (db/snapshot): Implement operation rollback (#2760)

* (db/snapshot): refactor Operations

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): refactor Operations

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add rollback for operations

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot) snapshot integrate (#2770)

* code opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add some APIs for SSDBImpl

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* partially add GetVectorById

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* snapshot opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix typo

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update GetVectorByID framework

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename GetResFiles to GetResPath

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update GetVectorByIdSegmentHandler

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add GetEntityByID

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update DataType

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update ParamField

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update GetEntityByIDSegmentHandler

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* support INT8 and INT16 in GetEntityById

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add GetIDsInSegment

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update FieldType

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add InsertEntities and DeleteEntities

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add HybridQuery

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* (db/snapshot): Add some helper visitors (#2777)

* (db/snapshot): add Segment related visitors

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* Update template

Signed-off-by: jinhai <hai.jin@zilliz.com>

* snapshot integrate (#2782)

* update HybridQuery

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add IterateHandler

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* IterateHandler opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rewrite mem manager 1 (#2762)

* rewrite mem manager

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* rewrite mem manager 2

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* refine

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* rewrite mem manager 3

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* merge manager 1

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix too many data copies (#2784)

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* fix a bug of getting wrong nprobe limitation in knowhere on GPU version (#2788)

* fix a bug of getting wrong nprobe limitation in knowhere on GPU version

Signed-off-by: cmli <chengming.li@zilliz.com>

* update changelog and fix build error

Signed-off-by: cmli <chengming.li@zilliz.com>

Co-authored-by: cmli <chengming.li@zilliz.com>

* Hnsw sq8 support (#2799)

* modify for SQ8

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* add hnsw-sq8-support

Signed-off-by: cmli <chengming.li@zilliz.com>

* make clang-format and update changelog

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix lint error

Signed-off-by: cmli <chengming.li@zilliz.com>

* hide unittest error temporary

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix ut error

Signed-off-by: cmli <chengming.li@zilliz.com>

Co-authored-by: shengjun.li <shengjun.li@zilliz.com>
Co-authored-by: cmli <chengming.li@zilliz.com>

* rewrite merge manager 1 (#2797)

* rewrite merge manager 1

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* rewrite merge manager 2

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* (db/snapshot): Add some visitors (#2800)

* (db/snapshot): add more visitors

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add some Iterators

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update visitors

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* fix #2752 (#2789)

* fix #2752

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>
(cherry picked from commit d7c47a7)

* #2802 Add new index: IVFSQ8NR (#2805)

* support new index ivfsq8nr

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* remove timeout

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* fix task_search (#2808)

* fix task_search

Signed-off-by: cqy <yaya645@126.com>

* fix unit test

Signed-off-by: cqy <yaya645@126.com>

* add unittest 4 hnsw_sq8nr and fix some bug in passing (#2819)

* modify for SQ8

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* add hnsw-sq8-support

Signed-off-by: cmli <chengming.li@zilliz.com>

* make clang-format and update changelog

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix lint error

Signed-off-by: cmli <chengming.li@zilliz.com>

* hide unittest error temporary

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix ut error

Signed-off-by: cmli <chengming.li@zilliz.com>

* add unittest 4 hnsw_sq8nr and fix several bugs

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix lint error

Signed-off-by: cmli <chengming.li@zilliz.com>

Co-authored-by: shengjun.li <shengjun.li@zilliz.com>
Co-authored-by: cmli <chengming.li@zilliz.com>

* snapshot integrate (#2814)

* update HybridQuery

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add IterateHandler

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* IterateHandler opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add SSSegmentReader

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add src/codecs/snapshot

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update latest code

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update SSSegmentReader interface

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add SSSegmentWriter

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update SSSegmentWriter interface

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest/db build fail

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* use SSSegmentReader and SSSegmentWriter in SSDBImpl

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix build issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* Add SDK support for IVFSQ8NR (#2821)

* add c++ sdk

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* fix clang

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* (db/snapshot): LSN logic change and design EventHandlerFactory (#2815)

* (db/snapshot): add more visitors

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add some Iterators

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update visitors

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add event handler factory

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): process lsn

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): small change

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): precheck duplicate collection in create collection operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* [skip ci] Update README

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* [skip ci] Update README_CN

Signed-off-by: JinHai-CN <hai.jin@zilliz.com>

* snapshot integrate (#2830)

* update HybridQuery

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add IterateHandler

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* IterateHandler opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* opt

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add SSSegmentReader

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add src/codecs/snapshot

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update latest code

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update SSSegmentReader interface

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add SSSegmentWriter

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update SSSegmentWriter interface

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest/db build fail

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* use SSSegmentReader and SSSegmentWriter in SSDBImpl

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix build issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add test_segment

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* (db/snapshot): update visitors (#2837)

* (db/snapshot): add more visitors

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add some Iterators

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update visitors

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add event handler factory

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): process lsn

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): small change

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): precheck duplicate collection in create collection operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update visitor

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update visitor 2

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update visitor 3

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* Fix rnsg master (#2831)

* fix rnsg ip

Signed-off-by: cqy <yaya645@126.com>

* fix rnsg search in master

Signed-off-by: cqy <yaya645@126.com>

* Hnsw sq8 support (#2835)

* modify for SQ8

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* add hnsw-sq8-support

Signed-off-by: cmli <chengming.li@zilliz.com>

* add unittest 4 hnsw_sq8nr and fix several bugs

Signed-off-by: cmli <chengming.li@zilliz.com>

Co-authored-by: shengjun.li <shengjun.li@zilliz.com>
Co-authored-by: cmli <chengming.li@zilliz.com>

* remove old api (#2833)

* remove old api

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* refine insert code

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* refine code

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* Update sqlite_6_to_4.sql (#2832)

Simplify SQL statements

Signed-off-by: dddddai <853238012@qq.com>

* (db/snapshot) Add snapshot DB module (#2851)

* New db impl

Signed-off-by: yhz <413554850@qq.com>

* Add db impl for snapshot

Signed-off-by: yhz <413554850@qq.com>

* Update operations step holders

Signed-off-by: yhz <413554850@qq.com>

* Update snapshot dbimpl & create collection test pass

Signed-off-by: yhz <413554850@qq.com>

* Delete operation pass

Signed-off-by: yhz <413554850@qq.com>

* Add constraints for db table

Signed-off-by: yhz <413554850@qq.com>

* Update index operation steps behavior

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Add lock in mysql engine and remove lock in store

Signed-off-by: yhz <413554850@qq.com>

* Index unittest case pass

Signed-off-by: yhz <413554850@qq.com>

* Add merge operation step context

Signed-off-by: yhz <413554850@qq.com>

* Remove constraints in DB and snapshot ut pass

Signed-off-by: yhz <413554850@qq.com>

* update DBImple

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Add DBHelper to generate sql & add mock Mem db & snaoshot ut pass

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Add mock db engine

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Update DB select API using template for resource

Signed-off-by: Yhz <yinghao.zou@zilliz.com>

* Update snapshot db name and move them under folder meta

Signed-off-by: yhz <413554850@qq.com>

* clang-format

Signed-off-by: yhz <413554850@qq.com>

* redefine SegmentWriter (#2853)

* redefine SegmentWriter

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* redefine SegmentWriter

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* refine code

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* move index type try2 (#2860)

* move EngineType and MetricType to MetaTypes.h

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move IndexType

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* snapshot integrate (#2854)

* add test_segment

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update interface GetResPath

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* retry ci

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update SSSegmentWriter

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* #2802 Add new index: IVFSQ8NR_NM (#2865)

* add ivfsq8nr_nm

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* refactor

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* fix error

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* [skip ci] remove unused param

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* (db/snapshot): Update row count related logic (#2862)

* (db/snapshot): update for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix bug in NewSegmentOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove dummy print

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add some test for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* add MILVUS_SUPPORT_SPTAG for some code (#2866)

* add MILVUS_SUPPORT_SPTAG for some code

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* use knowhere::IndexEnum in db

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* keep EngineType order

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* # 2828 opt faiss build (#2855)

* add FAISS_USE_FLOAT16

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix faiss build

Signed-off-by: cqy <yaya645@126.com>

Co-authored-by: yudong.cai <yudong.cai@zilliz.com>

* (db/snapshot): update size related logic (#2877)

* (db/snapshot): update for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix bug in NewSegmentOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove dummy print

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add some test for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update size logic

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update size logic related ut

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): rollback if operation is not done

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): clean store

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove some dependency

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot) Add snapshot DB unittest (#2856)

* (snapshot) Add snapshot DB unittest

Signed-off-by: yhz <413554850@qq.com>

* Optimize mock db

Signed-off-by: yhz <413554850@qq.com>

* correct server context file name

Signed-off-by: yhz <413554850@qq.com>

* Fix Prometheus rebuild problem (#2878)

* Solve prometheus repeatedly build problem

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* recover build.sh

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* refactor build.sh

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* altering changelog

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* rm not-useful comments

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>

* Add some cpp unittests about scheduler algorithm (#2888)

* add some testcases

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* reformat code

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
(cherry picked from commit 8135cee)

* opt faiss build  (#2883)

* opt faiss build

Signed-off-by: cqy <yaya645@126.com>

* use BlockingQueue in JobMgr (#2885)

* using BlockingQueue in JobMgr

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* segment writer (#2873)

* redefine SegmentWriter

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* redefine SegmentWriter

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* refine code

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* mem insert

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* unittest

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix some issues

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* segment reader

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* segment writer

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* remove files

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* remove code

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* merge

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix merge

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* get entities

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* Fixed broken links. (#2902)

Signed-off-by: PahudPlus <haimeng.cai@zilliz.com>

* Set MySQL host ip to 127.0.0.1 in jenkins unittest step (#2905)

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* Fix warning master (#2894)

* fix clean unused variable warning in knowhere

Signed-off-by: cqy <yaya645@126.com>

* clean unused variable warning in knowhere

Signed-off-by: cqy <yaya645@126.com>

* clean unused variable warning in knowhere

Signed-off-by: cqy <yaya645@126.com>

* clean unused variable warning in knowhere

Signed-off-by: cqy <yaya645@126.com>

* fix konwhere warning

Signed-off-by: cqy <yaya645@126.com>

* fix #2768 (#2809)

* fix #2752

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* fix #2768

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: groot <yihua.mo@zilliz.com>

* typo again

Signed-off-by: groot <yihua.mo@zilliz.com>

* [0.10.1] update shards ci && enable pq case (#2804)

* update cron on 0.10.1

Signed-off-by: zw <zw@milvus.io>

* enable ivf_pq <ip> search cases on GPU

Signed-off-by: zw <zw@milvus.io>

* [skip ci] debug

Signed-off-by: zw <zw@milvus.io>

* [skip ci] add artifacts

Signed-off-by: zw <zw@milvus.io>

* update shards ci && enable pq case

Signed-off-by: zw <zw@milvus.io>

* update assertion for pq

Signed-off-by: zw <zw@milvus.io>

* disable rnsg: issue #2813

Signed-off-by: zw <zw@milvus.io>

* add binary version in log file name

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* Add python functional tests about timeout (#2793)

* Add python tests about param timeout

Signed-off-by: yangxuan <jumpthepig@gmail.com>

* add faiss configure args msg print

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* change assertion conditions

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>

* add testcase of empty vector insert (#2850)

* add testcase of empty vector insert

Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>

* format python code

Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>

* (db/snapshot): Update Store (#2897)

* (db/snapshot): update for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix bug in NewSegmentOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove dummy print

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add some test for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update size logic

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update size logic related ut

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): rollback if operation is not done

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): clean store

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove some dependency

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update for store

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update Store.h

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update store related code

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add field element modification operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): change new operation name

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add AddSegmentFileOperation (#2924)

* (db/snapshot): update for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix bug in NewSegmentOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove dummy print

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add some test for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update size logic

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update size logic related ut

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): rollback if operation is not done

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): clean store

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove some dependency

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update for store

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update Store.h

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update store related code

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add field element modification operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): change new operation name

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add Segment File Operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* fix: upgrade com.alibaba:fastjson from 1.2.71 to 1.2.72 (#2925)

Snyk has created this PR to upgrade com.alibaba:fastjson from 1.2.71 to 1.2.72.

See this package in NPM:
https://www.npmjs.com/package/com.alibaba:fastjson

See this project in Snyk:
https://app.snyk.io/org/cxie/project/2f701ce6-b421-4bbf-aaad-9d2bdb8569e4?utm_source=github&utm_medium=upgrade-pr

* (db/snapshot) Optimize snapshot mock db (#2896)

* Add mysql time statices

Signed-off-by: yhz <413554850@qq.com>

* Add detail timestamp recorder

Signed-off-by: yhz <413554850@qq.com>

* Optimize snapshot mock DB

Signed-off-by: yhz <413554850@qq.com>

* Remove surplus space

Signed-off-by: yhz <413554850@qq.com>

* Updat meta test

Signed-off-by: yhz <413554850@qq.com>

* Rearrange core cmakelists (#2900)

* sort out cmake commands in before project

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* update

Signed-off-by: Xiangyu Wang <xy.wang@zilliz.com>

* rearrange core CmakeLists.txt

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix conflict

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix bugs on ENABLE_FIU

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: Xiangyu Wang <xy.wang@zilliz.com>

* add web_impl testcase (#2920)

Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>

* [skip ci] Update cases for the resources changed (#2907)

* update cron on 0.10.1

Signed-off-by: zw <zw@milvus.io>

* enable ivf_pq <ip> search cases on GPU

Signed-off-by: zw <zw@milvus.io>

* [skip ci] debug

Signed-off-by: zw <zw@milvus.io>

* [skip ci] add artifacts

Signed-off-by: zw <zw@milvus.io>

* update shards ci && enable pq case

Signed-off-by: zw <zw@milvus.io>

* update assertion for pq

Signed-off-by: zw <zw@milvus.io>

* disable rnsg: issue #2813

Signed-off-by: zw <zw@milvus.io>

* add binary version in log file name

Signed-off-by: zw <zw@milvus.io>

* enable rnsg case

Signed-off-by: zw <zw@milvus.io>

* [skip ci] Update config cases for the resources changed

Signed-off-by: zw <zw@milvus.io>

* [skip ci] Fix limit

Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>

* refine code (#2927)

* refine code

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* excution engine

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* (db/snapshot): add GC event and implement crtp for base resource (#2926)

* (db/snapshot): update for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix bug in NewSegmentOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove dummy print

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add some test for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update size logic

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update size logic related ut

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): rollback if operation is not done

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): clean store

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove some dependency

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update for store

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update Store.h

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update store related code

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add field element modification operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): change new operation name

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add Segment File Operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): crtp for BaseResource

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add InActiveResourcesGCEvent

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix ut error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): small change

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot) Add root path for snapshot GC (#2936)

* Add root path for snapshot GC

Signed-off-by: yhz <413554850@qq.com>

* Add subfolder db in root path

Signed-off-by: yhz <413554850@qq.com>

* #2890 Fix wrong index size (#2930)

* fix update index size

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* add GetSize() interface 4 SPTAG

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix index size of index hnsw, annoy and nsg

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix clang

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* add sdk 4 non-ivf index test

Signed-off-by: cmli <chengming.li@zilliz.com>

* remove std::cout and make clang-format

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix cout

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* checkout sdk

Signed-off-by: cmli <chengming.li@zilliz.com>

* fix binary ivf

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* fix hnsw

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

Co-authored-by: cmli <chengming.li@zilliz.com>
Co-authored-by: shengjun.li <shengjun.li@zilliz.com>

* scheduler support snapshot (#2944)

* using BlockingQueue in JobMgr

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update changelog

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update SSDBImpl::HybridQuery

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add QueryTest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add SearchJob interface

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add SSSearchJob and SSSearchTask

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update AddSegmentVisitor

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add SSBuildIndexTask and SSBuildIndexJob

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add SSTestTask

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add test_ss_job and test_ss_task

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix clang-format

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update TaskCreator

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* #2890 Fix wrong index size (#2942)

* fix NM index size

Signed-off-by: sahuang <xiaohai.xu@zilliz.com>

* add raw data size of hnsw_nm and nsg_nm

Signed-off-by: cmli <chengming.li@zilliz.com>

Co-authored-by: cmli <chengming.li@zilliz.com>

* Minor editorial updates. (#2946)

Signed-off-by: Amy Hong <yun.hong@zilliz.com>

* fix cuda version requirement (#2909)

Signed-off-by: Xiangyu Wang <xy.wang@zilliz.com>

* snapshot scheduler (#2961)

* update interface

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add test_ss_job

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update Query

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add dir_root for SSSearchJob and SSBuildIndexJob

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* (db/snapshot): Update DropAllIndexOperation (#2947)

* (db/snapshot): update for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix bug in NewSegmentOperation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove dummy print

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add some test for row count

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update size logic

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update size logic related ut

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): rollback if operation is not done

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): clean store

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): remove some dependency

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update for store

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update Store.h

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update store related code

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add field element modification operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): change new operation name

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): Add Segment File Operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): crtp for BaseResource

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): add InActiveResourcesGCEvent

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix ut error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): small change

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update snapshot segmentcommit operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update drop all index operation

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): update ut

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* (db/snapshot): fix lint error

Signed-off-by: peng.xu <peng.xu@zilliz.com>

* fix a bug of annoy search parameter exisitence check (#2958)

Signed-off-by: cmli <chengming.li@zilliz.com>

Co-authored-by: cmli <chengming.li@zilliz.com>

* (db/snapshot) Add root path for snapshot store (#2945)

* Add root path for snapshot GC

Signed-off-by: yhz <413554850@qq.com>

* Change Fun name from GetResPath to GetRootPath

Signed-off-by: yhz <413554850@qq.com>

* Fix the result merging of IVF_PQ IP (#2955)

Signed-off-by: shengjun.li <shengjun.li@zilliz.com>

* Fix C++ sdk DataType

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix collection test cases bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* (db/snapshot) Inactivate gc event (#2960)

* Add API to Get inactivate res in Store

Signed-off-by: yhz <413554850@qq.com>

* Update select logic in MetaAdapter

Signed-off-by: yhz <413554850@qq.com>

* Add event test case

Signed-off-by: yhz <413554850@qq.com>

* Add deactive collection test in SSEventTest.TestInActiveResGcEvent

Signed-off-by: yhz <413554850@qq.com>

* Add CollectinCommit in event test

Signed-off-by: yhz <413554850@qq.com>

* Add more res in event test

Signed-off-by: yhz <413554850@qq.com>

* Set url to mock in ssdb test

Signed-off-by: yhz <413554850@qq.com>

* snapshot build index (#2939)

* refine code

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* refine

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* ui error

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* build index

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* add utils

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* build index

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* build index

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* execution engine

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* typo

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* build error

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* Add options to build.sh for cuda arch Issue#2940 (#2950)

* add option -s in build.sh for cuda arch

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* modify changelog

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* modify ci configs about cuda arch

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix bug

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* modify ci configs about cuda arch in jenkins ci

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* fix bug

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* fix bug

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* modify changelog

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix bug

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* fix bug

Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>

* fix bugs

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix cmake cannot recognize space bug

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix conflict

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: quicksilver <zhifeng.zhang@zilliz.com>

* Fix test_rpc bugs

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix test_rpc

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: op-hunter <ophunter52@gmail.com>
Co-authored-by: cmli <chengming.li@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: BossZou <40255591+BossZou@users.noreply.github.com>
Co-authored-by: Xiaohai Xu <xiaohai.xu@zilliz.com>
Co-authored-by: shengjun.li <shengjun.li@zilliz.com>
Co-authored-by: XuPeng-SH <xupeng3112@163.com>
Co-authored-by: quicksilver <zhifeng.zhang@zilliz.com>
Co-authored-by: groot <yhmo@zeronedata.com>
Co-authored-by: groot <yihua.mo@zilliz.com>
Co-authored-by: Jin Hai <hai.jin@zilliz.com>
Co-authored-by: del-zhenwu <56623710+del-zhenwu@users.noreply.github.com>
Co-authored-by: zw <zw@milvus.io>
Co-authored-by: Snyk bot <snyk-bot@snyk.io>
Co-authored-by: cqy123456 <39671710+cqy123456@users.noreply.github.com>
Co-authored-by: dwq <41563853+dddddai@users.noreply.github.com>
Co-authored-by: XuanYang-cn <51370125+XuanYang-cn@users.noreply.github.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: PahudPlus <64403786+PahudPlus@users.noreply.github.com>
Co-authored-by: godchen0212 <67679556+godchen0212@users.noreply.github.com>
Co-authored-by: Xiangyu Wang <xy.wang@zilliz.com>
Co-authored-by: AmyYH <68527082+AmyYH@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Issues or changes related a bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant