Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[pull] master from milvus-io:master #8

Open
wants to merge 1,687 commits into
base: master
Choose a base branch
from
Open

[pull] master from milvus-io:master #8

wants to merge 1,687 commits into from
This pull request is big! We’re only showing the most recent 250 commits.

Commits on Jun 12, 2024

  1. fix: Prevent restart timetick sender creating ut datanode (#33790) 

    See also #33789

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    9ab3058 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. enhance: Avoid merging insert data when buffering insert msgs (#33562) 

    See also #33561

This PR:
- Use zero copy when buffering insert messages
- Make `storage.InsertCodec` support serialize multiple insert data
chunk into same batch binlog files

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    512ea6b View commit details
    Browse the repository at this point in the history

  2. test: add cases for go milvusclient delete (#33777) 

    issue: #33419

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
    @ThreadDao
    ThreadDao committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    9e99f96 View commit details
    Browse the repository at this point in the history

  3. fix: update clang-tidy and clang-format from 10 to 12 (#33141) 

    Default llvm toolchain version in Ubuntu 20.04 is 10, while Ubuntu 22.04
does not have `clang-tidy-10` or `clang-format-10` by default.

issue: #33142

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
    @jiangyinzuo
    jiangyinzuo committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    7d74edd View commit details
    Browse the repository at this point in the history

  4. fix: [skip e2e] Skip unstable integration test for master (#33824) 

    See also #33716 #33823

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    144ee26 View commit details
    Browse the repository at this point in the history

  5. test: update import test case to support different dim (#33709) 

    add test case for #33681

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
    @zhuwenxing
    zhuwenxing committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    ca1f7ab View commit details
    Browse the repository at this point in the history

  6. doc: [skip-e2e] add comments for meta table (#33403) 

    Add commnets for meta table in internal/rootcoord/meta_table.go.

Signed-off-by: ArenaSu <704427617@qq.com>
    @ArenaSu
    ArenaSu committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    db59249 View commit details
    Browse the repository at this point in the history

  7. doc: [skip-e2e] add comments for event manager (#33444) 

    Add comments for event manager(internal/datanode/event_manager.go).

Signed-off-by: ArenaSu <704427617@qq.com>
    @ArenaSu
    ArenaSu committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    2dfa752 View commit details
    Browse the repository at this point in the history

  8. enhance: Use BatchPkExist to reduce bloom filter func call cost (#33611) 

    issue:#33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    ab93d9c View commit details
    Browse the repository at this point in the history

  9. fix: Fix meta prefix overlap bug (#33830) 

    #30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    08fcf3f View commit details
    Browse the repository at this point in the history

  10. fix: Remove eviction operations out of lock (#33834) 

    See also #33823

`EvictCacheValueByFormat` may be block by on going `CASCacheValue` and
cause possible deadlock

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    c689ef4 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. Update all contributors 

    Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
    @sre-ci-robot
    sre-ci-robot committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    f3e4cb0 View commit details
    Browse the repository at this point in the history

  2. fix: Fallback field type when it isn't in request (#33832) 

    issue: #33432

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
    @xiaocai2333
    xiaocai2333 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    b69e909 View commit details
    Browse the repository at this point in the history

  3. enhance: Execute bloom filter apply in parallel to speed up segment p… 

    …redict (#33792)

issue: #33610

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    4987067 View commit details
    Browse the repository at this point in the history

  4. enhance: legacy code clean up (#33838) 

    issue: #33839

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
    @shaoting-huang
    shaoting-huang committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    0ecd694 View commit details
    Browse the repository at this point in the history

  5. enhance: Tidy compaction executor (#33778) 

    Move compaction executor to compaction pacakge.

issue: #32451

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
    @bigsheeper
    bigsheeper committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    86a36b1 View commit details
    Browse the repository at this point in the history

  6. fix: load operation when segment is on releasing (#31340) 

    issue: #30857

---------

Signed-off-by: chyezh <chyezh@outlook.com>
    @chyezh
    chyezh committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    9b69601 View commit details
    Browse the repository at this point in the history

  7. test: refine load state check in restful test (#33861) 

    Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
    @zhuwenxing
    zhuwenxing committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    a963afa View commit details
    Browse the repository at this point in the history

  8. test: add different datatype for some functions (#33869) 

    Signed-off-by: nico <cheng.yuan@zilliz.com>
    @NicoYuan1986
    NicoYuan1986 committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    b748d8a View commit details
    Browse the repository at this point in the history

  9. fix: DeleteData merge wrong data casuing data loss (#33820) 

    See also: #33819

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
    @XuanYang-cn
    XuanYang-cn committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    f67b6dc View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2024

  1. fix: readd timestamp index because segment timestamp not ordered (#33856 

    )

#33533

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
    @zhagnlu
    zhagnlu and luzhang committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    e422168 View commit details
    Browse the repository at this point in the history

  2. enhance: support array bitmap index (#33527) 

    #32900

---------

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
    @zhagnlu
    zhagnlu and luzhang committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    d43ec4d View commit details
    Browse the repository at this point in the history

  3. enhance: Rename Compaction to CompactionV2 (#33858) 

    Due to the removal of injection and syncSegments from the compaction, we
need to ensure that no compaction is successfully executed during the
rolling upgrade. This PR renames Compaction to CompactionV2, with the
following effects:
- New datacoord + old datanode: Utilizes the CompactionV2 interface,
resulting in the datanode error "CompactionV2 not implemented," causing
compaction to fail;
- Old datacoord + new datanode: Utilizes the CompactionV1 interface,
resulting in the datanode error "CompactionV1 not implemented," causing
compaction to fail.

issue: #32809

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
    @bigsheeper
    bigsheeper committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    8537f3d View commit details
    Browse the repository at this point in the history

  4. enhance: Ensure the idempotency of compaction task (#33872) 

    /kind enhancement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
    @bigsheeper
    bigsheeper committed Jun 16, 2024
    Configuration menu
    Copy the full SHA
    1a9ab52 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2024

  1. enhance: Add rbac support on describe/alter database api (#33803) 

    issue: #32707

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    b72026c View commit details
    Browse the repository at this point in the history

  2. fix: Don't remove growing L0 segment in datanode metacache (#33829) 

    issue: #33540 
1. gorwing L0 segments is invisible to datacoord.
2. flushed L0 segments need to clean by datacoord.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
    @xiaocai2333
    xiaocai2333 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    9514886 View commit details
    Browse the repository at this point in the history

  3. enhance: Refine name rule check error msg (#33815) 

    Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    188ee43 View commit details
    Browse the repository at this point in the history

  4. enhance: Reserve space of payload writer when serialize data (#33817) 

    See also #33561 #33562

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    f993b29 View commit details
    Browse the repository at this point in the history

  5. enhance: add Slice func for go milvusclient column (#33836) 

    issue: #33419

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
    @ThreadDao
    ThreadDao committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    2da289c View commit details
    Browse the repository at this point in the history

  6. fix: Prevent use captured iteration variable partitionID (#33906) 

    See also #33902

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    2a04b09 View commit details
    Browse the repository at this point in the history

  7. Update all contributors 

    Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
    @sre-ci-robot
    sre-ci-robot committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    88cca9d View commit details
    Browse the repository at this point in the history

  8. enhance: fix not find arrow::parquet lib (#33928) 

    Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
    @Presburger
    Presburger committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    e83ecd5 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. fix: fix data codec writer close (#33818) 

    issue:#33813

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
    @shaoting-huang
    shaoting-huang committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    8cdc0e6 View commit details
    Browse the repository at this point in the history

  2. fix: Check nodeID wildcard when removing pkOracle (#33895) 

    See also #33894

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    ec64499 View commit details
    Browse the repository at this point in the history

  3. enhance: growing segment support mmap (#32633) 

    issue: #32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
    @cqy123456
    cqy123456 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    32f685f View commit details
    Browse the repository at this point in the history

  4. enhance: Force to reset coord connection for unavailable error (#33908) 

    Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    5e906fd View commit details
    Browse the repository at this point in the history

  5. fix: Check the correct return error in MultiRemove (#33926) 

    See also #33925

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    5772123 View commit details
    Browse the repository at this point in the history

  6. fix: Return record with largest timestamp for entires with same PK (#… 

    …33936)

See also #33883

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    3fdaae8 View commit details
    Browse the repository at this point in the history

  7. fix: Small fixs of major compaction (#33929) 

    #30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    5cb0760 View commit details
    Browse the repository at this point in the history

  8. fix: can't find Chunk struct after growing support mmap (#33951) 

    issue: #32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
    @cqy123456
    cqy123456 committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    b460862 View commit details
    Browse the repository at this point in the history

  9. Update all contributors 

    Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
    @sre-ci-robot
    sre-ci-robot committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    7418609 View commit details
    Browse the repository at this point in the history

  10. enhance: autoindex for multi data type (#33868) 

    issue: #22837 

contain #33625
#33867
#33911 which already merged to
2.4 branch

Signed-off-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: foxspy <xianliang.li@zilliz.com>
    @chasingegg @foxspy
    chasingegg and foxspy committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    a789c60 View commit details
    Browse the repository at this point in the history

  11. fix: fix binary vector data size (#33750) 

    issue: #22837

- fix byte size wrong for binary vectors
- fix the expect/actual error msg

Signed-off-by: chasingegg <chao.gao@zilliz.com>
    @chasingegg
    chasingegg committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    0d20303 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. enhance: [skip e2e] modify gc configuration document (#33946) 

    issue: #31740

Signed-off-by: chyezh <chyezh@outlook.com>
    @chyezh
    chyezh committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    2f6f964 View commit details
    Browse the repository at this point in the history

  2. enhance: check index with data type (#33880) 

    issue: #22837
related: #33878

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
    @cqy123456
    cqy123456 committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    298e50b View commit details
    Browse the repository at this point in the history

  3. enhance: Avoid to iterate whole segment list for each task's process (#… 

    …33943)

when querycoord process segment task, it will try to iterate whole
segment list to checke whether segment is loaded, which cost too much
cpu if there has thousands of segments.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    0294595 View commit details
    Browse the repository at this point in the history

  4. fix: Panic if ProcessActiveStandBy returns error (#33369) 

    #33368

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    5fac2fa View commit details
    Browse the repository at this point in the history

  5. enhance: fix copying hits of inverted index twice (#33968) 

    issue: #29793
The custom `VecCollector` have already transformed the results into
vector of offsets, no need to copy them twice.

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
    @longjiquan
    longjiquan committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    7b9462c View commit details
    Browse the repository at this point in the history

  6. feat: adding deltalog stream reader and writer (#33844) 

    See #31679

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
    @tedxu
    tedxu committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    6d5747c View commit details
    Browse the repository at this point in the history

  7. test: Add more sparse test cases (#33916) 

    issue: #31483

Signed-off-by: elstic <hao.wang@zilliz.com>
    @elstic
    elstic committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    1216a4b View commit details
    Browse the repository at this point in the history

  8. enhance: update amazonlinux2023 image builder (#33948) 

    #32982

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
    @shaoting-huang
    shaoting-huang committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    eb3197e View commit details
    Browse the repository at this point in the history

  9. enhance: support null in go payload (#32296) 

    #31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
    @smellthemoon
    smellthemoon and lixinguo committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    2a13569 View commit details
    Browse the repository at this point in the history

  10. fix: Remove loopclosure issue in ChannelManagerImplV2 (#33989) 

    See also #33987

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    6a15b6e View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2024

  1. enhance: Set maxPartitionNum default value to 1024 (#33949) 

    See also #30059

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    cc77363 View commit details
    Browse the repository at this point in the history

  2. enhance: update pymilvus version (#33905) 

    Signed-off-by: nico <cheng.yuan@zilliz.com>
    @NicoYuan1986
    NicoYuan1986 committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    0264588 View commit details
    Browse the repository at this point in the history

  3. test: add test case for bulkwriter (#33879) 

    Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
    @zhuwenxing
    zhuwenxing committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    f3d902c View commit details
    Browse the repository at this point in the history

  4. enhance: Add trace for bf cost in l0 compactor (#33860) 

    Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    31ef0a1 View commit details
    Browse the repository at this point in the history

  5. enhance: Refactor data codec deserialize (#33923) 

    #33922

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
    @shaoting-huang
    shaoting-huang committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    5f02e52 View commit details
    Browse the repository at this point in the history

  6. test: change the number of insertion vectors (#34014) 

    pr: #33916 

1. Fix case: test_collection_multi_sparse_vectors 
2. Reduce the amount of data generated

Signed-off-by: elstic <hao.wang@zilliz.com>
    @elstic
    elstic committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    ce3d707 View commit details
    Browse the repository at this point in the history

  7. fix: Fix GetReplicas API return nil status (#33715) 

    issue: #33702

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    935bc1f View commit details
    Browse the repository at this point in the history

  8. enhance: upgrade build-env to ubuntu 22.04 and gcc12 (#33959) 

    issue: #33960

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
    @yellow-shine
    yellow-shine committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    5be9929 View commit details
    Browse the repository at this point in the history

  9. enhance: use segment id and type to register in MmapChunkManager and … 

    …opt malloc in variableChunk (#33993)

issue: #32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
    @cqy123456
    cqy123456 committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    dc4437f View commit details
    Browse the repository at this point in the history

  10. enhance: Add deltaRowCount in l0 compaction (#33997) 

    See also: #33998

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
    @XuanYang-cn
    XuanYang-cn committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    04edb07 View commit details
    Browse the repository at this point in the history

  11. fix: fix milvus gpu compile error (#33995) 

    NVIDIA RAFT uses customized versions of spdlog and fmt, which conflict
with the versions provided by Conan. Therefore, it is necessary to
isolate the build environment for knowhere to avoid these conflicts.
issue: #33991

Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
    @Presburger
    Presburger committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    dac20d4 View commit details
    Browse the repository at this point in the history

  12. [automated] Update Knowhere Commit (#34040) 

    Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
    @sre-ci-robot @github-actions
    sre-ci-robot and github-actions[bot] committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    d9f9864 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2024

  1. test: optimizing variable names (#34035) 

    Signed-off-by: elstic <hao.wang@zilliz.com>
    @elstic
    elstic committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    4e414fb View commit details
    Browse the repository at this point in the history

  2. enhance: Add rules and fix for go_client e2e code style (#34033) 

    See also #31293

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    35ea775 View commit details
    Browse the repository at this point in the history

  3. enhance: Support queryOption WithLimit for milvusclient (#33978) 

    See also #31293

---------

Signed-off-by: coldWater <254244460@qq.com>
    @forsaken628
    forsaken628 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    6671000 View commit details
    Browse the repository at this point in the history

  4. fix: metrics database_num is 0 after restarting rootcoord (#34011) 

    issue: #34041

Signed-off-by: jaime <yun.zhang@zilliz.com>
    @jaime0815
    jaime0815 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    e653ad2 View commit details
    Browse the repository at this point in the history

  5. test: update the lib of bf16 (#34043) 

    Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
    @zhuwenxing
    zhuwenxing committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    03a5f7e View commit details
    Browse the repository at this point in the history

  6. feat: adding grouped virtual resource allocator (#33669) 

    See #33559

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
    @tedxu
    tedxu committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    958ecd5 View commit details
    Browse the repository at this point in the history

  7. fix: fix l0 compaction may miss some sealed segments (#33980) 

    issue: #33955

Signed-off-by: sunby <sunbingyi1992@gmail.com>
    @sunby
    sunby committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    622be36 View commit details
    Browse the repository at this point in the history

  8. enhance: Unify DeleteLog parsing code (#34009) 

    See also #33787

The parsing delete log is distributed in lots of places, which is not
recommended and hard to maintain.

This PR abstract common parsing logic into `DeleteLog.Parse` method to
unify implementation and make it easier to replace json parsing lib.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    2f691f1 View commit details
    Browse the repository at this point in the history

  9. fix: Fix memory buffer error & some renaming (#33850) 

    #30633

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    380d3f4 View commit details
    Browse the repository at this point in the history

  10. test: Update tests for range search and add test for query with dup i… 

    …ds (#34057)

related issue: #33883

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
    @yanliang567
    yanliang567 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    f3f390e View commit details
    Browse the repository at this point in the history

  11. test: modify test case about upsert partition (#33919) 

    issue: #33902

Signed-off-by: nico <cheng.yuan@zilliz.com>
    @NicoYuan1986
    NicoYuan1986 committed Jun 21, 2024
    Configuration menu
    Copy the full SHA
    c85644e View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2024

  1. enhance: async search and retrieve in cgo (#33228) 

    issue: #30926, #33132
related pr: #33133

---------

Signed-off-by: chyezh <chyezh@outlook.com>
    @chyezh
    chyezh committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    259a682 View commit details
    Browse the repository at this point in the history

  2. enhance: [skip e2e] Update mergify.yml to skip cov checker (#34072)

    @zhikunyao
    zhikunyao committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    e0e06fc View commit details
    Browse the repository at this point in the history

  3. fix: Do compressBinlog to fix logID 0 (#34060) 

    issue: #34059

Do compressBinlog to ensure that reloadFromKV will fill binlogs' logID
after datacoord restarts.

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
    @bigsheeper
    bigsheeper committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    fb870d2 View commit details
    Browse the repository at this point in the history

  4. Update all contributors 

    Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
    @sre-ci-robot
    sre-ci-robot committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    e1b64e2 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2024

  1. enhance: get environment variable from .env (#34081) 

    Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
    @yellow-shine
    yellow-shine committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    b77ab76 View commit details
    Browse the repository at this point in the history

  2. enhance: Remove the unused compaction logic from shard (#33932) 

    1. Remove the `compactTo` field in `SegmentInfo`.
2. Remove the target segment not match and its retry logic in
`SyncManager`.

issue: #32809

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
    @bigsheeper
    bigsheeper committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    6c1d815 View commit details
    Browse the repository at this point in the history

  3. enhance: Enhance and correct exception module (#33705) 

    #33704

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
    @zhagnlu
    zhagnlu and luzhang committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    0d7ea8e View commit details
    Browse the repository at this point in the history

  4. enhance: add skip using array index when some situation (#33947) 

    #32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
    @zhagnlu
    zhagnlu and luzhang committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    03a3f50 View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2024

  1. fix: turn on compression on stream writers (#34067) 

    See #31679

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
    @tedxu
    tedxu committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    78885a4 View commit details
    Browse the repository at this point in the history

  2. fix: sync part stats task cannot be finished(#30376) (#34027) 

    related: #30376
also: refine log output for query_coord task by rephrasing action string

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
    @MrPresent-Han
    MrPresent-Han and MrPresent-Han committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    ca7ef26 View commit details
    Browse the repository at this point in the history

  3. fix: Prevent remove new growing L1 segment when SyncSegments (#34056) 

    Related to #34018

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    b5c9a73 View commit details
    Browse the repository at this point in the history

  4. enhance: support integral type for MV and skip MV if there is only on… 

    …e category (#33161)

issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
    @PwzXxm
    PwzXxm committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    b961767 View commit details
    Browse the repository at this point in the history

  5. enhance: wal interface definition (#33745) 

    issue: #33285

---------

Signed-off-by: chyezh <chyezh@outlook.com>
    @chyezh
    chyezh committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    b923728 View commit details
    Browse the repository at this point in the history

  6. test: refine restful testcases trace (#34066) 

    Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
    @zhuwenxing
    zhuwenxing committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    4c6f6c5 View commit details
    Browse the repository at this point in the history

  7. enhance: enable flush rate limiter of collection level (#33837) 

    Signed-off-by: jaime <yun.zhang@zilliz.com>
    @jaime0815
    jaime0815 committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    d08cb88 View commit details
    Browse the repository at this point in the history

  8. fix: [ut] regex query under unsupported index (#34087) 

    /kind improvement
issue: #29988

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
    @longjiquan
    longjiquan committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    aa36f9f View commit details
    Browse the repository at this point in the history

  9. fix: update ubuntu base image version (#33944) 

    related to #33945
FIX CVEs of milvus base image: MEDIUM: 8, **Total FIX: 8** for master
branch

![image](https://github.com/milvus-io/milvus/assets/27683687/71708028-265e-4f20-8732-56b7e76864fc)

Signed-off-by: weizhenkun <zhenkun.wei@qq.com>
Co-authored-by: weizhenkun <zhenkun.wei@qq.com>
    @weiZhenkun
    weiZhenkun and weizhenkun committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    a1248a1 View commit details
    Browse the repository at this point in the history

  10. enhance: Add nilness linter and fix some small issues (#34049) 

    Add `nilness` for govet linter and fixed some detected issues

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    fd922d9 View commit details
    Browse the repository at this point in the history

  11. enhance: [skip e2e]Update mergify.yml to skip code check workflow (#3… 

    …4113)

Signed-off-by: Zhikun Yao <zhikun.yao@zilliz.com>
Co-authored-by: Zhikun Yao <zhikun.yao@zilliz.com>
    @zhikunyao
    zhikunyao and Zhikun Yao committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    68fe6f9 View commit details
    Browse the repository at this point in the history

  12. fix: Make querycoord panick when rg metastore sync fail (#34106) 

    See also #34047

When `unassignNode` sync resource group with node removed failed

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    07c25a1 View commit details
    Browse the repository at this point in the history

  13. [automated] Update Knowhere Commit (#34119) 

    Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
    @sre-ci-robot @github-actions
    sre-ci-robot and github-actions[bot] committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    254b555 View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. fix: Deep copy ImportTask.segmentsInfo to prevent data race (#34090) 

    See also #34089

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    506a915 View commit details
    Browse the repository at this point in the history

  2. enhance: Refine compaction (#33982) 

    issue : #32939

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
    @czs007
    czs007 committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    d18c490 View commit details
    Browse the repository at this point in the history

  3. [automated] Update cpu Builder image changes (#34078) 

    Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
    @yellow-shine
    yellow-shine committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    2cc9df5 View commit details
    Browse the repository at this point in the history

  4. enhance: [skip e2e] Check 'Build and Test' steps success for mergify (#… 

    …34125)

Since 'Build and test' action is separated into several steps, each
failure/success status shall cause different behavior for mergify rule

This PR check cpp/go ut and intergration tests for 'Build & test'
actions to add or remove labels.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    a21c086 View commit details
    Browse the repository at this point in the history

  5. enhance: revert dead try on uploading codecov (#34130) 

    This reverts commit 19936d9.

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
    @yellow-shine
    yellow-shine committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    ac28402 View commit details
    Browse the repository at this point in the history

  6. fix: add build-tags at golangci-lint (#34084) 

    issue: #33132

caused by introducing conditional compile file in async cgo.

Signed-off-by: chyezh <chyezh@outlook.com>
    @chyezh
    chyezh committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    93291c0 View commit details
    Browse the repository at this point in the history

  7. enhance: Add ctx in SyncTask.Run to be cancellable (#34042) 

    Related to #33716

This PR add context param in SyncTask.Run execution functions to make it
cancellable from the caller.

This make it possible to cancel task when datanode/data sync service is
beeing shut down.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    962a544 View commit details
    Browse the repository at this point in the history

  8. doc: [skip e2e] change milvus docker image version to v2.4.5 (#34135) 

    Signed-off-by: ping.liu <ping.liu@zilliz.com>
    @pingliu
    pingliu committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    b12c34a View commit details
    Browse the repository at this point in the history

  9. fix: Prevent import from generating orphaned files (#34070) 

    When import failed, mark the import segment as dropped instead of drop
it directly to prevent generating orphaned files.

issue: #34068

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
    @bigsheeper
    bigsheeper committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    940a0ac View commit details
    Browse the repository at this point in the history

  10. enhance: [skip e2e] Sync mergify Build & test job name to 22.04 (#34158) 

    Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    c8c423e View commit details
    Browse the repository at this point in the history

  11. Add an option to enable/disable vector field clustering key (#34097) 

    #30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    f9a0f7b View commit details
    Browse the repository at this point in the history

  12. fix: [GoSDK] Compose result set for result zero hit case (#34129) 

    See also #33952

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    b1a3473 View commit details
    Browse the repository at this point in the history

  13. fix: Only sync flushed segments to datanode (#34156) 

    issue: #33540

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
    @xiaocai2333
    xiaocai2333 committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    c65f41d View commit details
    Browse the repository at this point in the history

  14. enhance: move rocksmq from internal to pkg module (#33881) 

    issue: #33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
    @jaime0815
    jaime0815 committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    9630974 View commit details
    Browse the repository at this point in the history

  15. enhance: update new mmap config parmeters version (#34143) 

    issue: #32984
related pr: #34145

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
    @cqy123456
    cqy123456 committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    381d437 View commit details
    Browse the repository at this point in the history

  16. [automated] Update Knowhere Commit (#34172) 

    Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
    @sre-ci-robot @github-actions
    sre-ci-robot and github-actions[bot] committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    cc54ab2 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. test: add cases for go milvusclient query (#33884) 

    issue: #33419

---------

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
    @ThreadDao
    ThreadDao committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    a77e295 View commit details
    Browse the repository at this point in the history

  2. fix: fix error ignore in compactor (#34169) 

    #34170

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    fbe3231 View commit details
    Browse the repository at this point in the history

  3. fix: centroids file not removed when data skew in major compaction (#… 

    …34050)

issue: #30633

Signed-off-by: chasingegg <chao.gao@zilliz.com>
    @chasingegg
    chasingegg committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    a933f67 View commit details
    Browse the repository at this point in the history

  4. enhance: Avoid search querynode return nil status in response (#34100) 

    Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    4520342 View commit details
    Browse the repository at this point in the history

  5. fix: Fix datacoord's properties overwriten (#34146) 

    issue: #34144 
/kind bug

Signed-off-by: xige-16 <xige2016@gmail.com>
    @xige-16
    xige-16 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    f8bf690 View commit details
    Browse the repository at this point in the history

  6. enhance:[skip e2e]update one pod mode resource (#34196) 

    Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
    @zhuwenxing
    zhuwenxing committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    a08000c View commit details
    Browse the repository at this point in the history

  7. enhance: Add unittest for storage.DeleteLog (#34190) 

    See also #33787
Backport unit test part in #34188

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    e04f1f9 View commit details
    Browse the repository at this point in the history

  8. enhance: [skip e2e] make mergify back-forward compatible with old che… 

    …ck (#34183)

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
    @yellow-shine
    yellow-shine committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    0565b70 View commit details
    Browse the repository at this point in the history

  9. enhance: [skip e2e] fix mergify (#34209) 

    Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
    @yellow-shine
    yellow-shine committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    e43c558 View commit details
    Browse the repository at this point in the history

  10. fix: Pass otlpSecure config when setup segcore tracing (#34193) 

    See also #34192

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    948bfba View commit details
    Browse the repository at this point in the history

  11. Update all contributors 

    Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
    @sre-ci-robot
    sre-ci-robot committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    006f753 View commit details
    Browse the repository at this point in the history

  12. enhance: remove CheckVecIndexWithDataTypeExist function in pkg and re… 

    …move some cgo call (#34102)

issue: #22837
related pr: #34104

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
    @cqy123456
    cqy123456 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    51ebe95 View commit details
    Browse the repository at this point in the history

  13. enhance: refine error message for invalid limit parameter (#34178) 

    Signed-off-by: jaime <yun.zhang@zilliz.com>
    @jaime0815
    jaime0815 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    9c2eeff View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. test: add cases got gosdk v2 search and databases (#34204) 

    issue: #33952 #33627 #33419

---------

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
    @ThreadDao
    ThreadDao committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    be23495 View commit details
    Browse the repository at this point in the history

  2. enhance: implement rmq and pulsar as wal (#34046) 

    issue: #33285

- use reader but not consumer for pulsar
- advanced test framework
- move some streaming related package into pkg

---------

Signed-off-by: chyezh <chyezh@outlook.com>
    @chyezh
    chyezh committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    d2bc4a5 View commit details
    Browse the repository at this point in the history

  3. test: update test cases about autoindex (#34224) 

    Signed-off-by: nico <cheng.yuan@zilliz.com>
    @NicoYuan1986
    NicoYuan1986 committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    f2107ad View commit details
    Browse the repository at this point in the history

  4. enhance: Update knowhere commit (#34223) 

    Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
    @cydrain
    cydrain committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    ad90360 View commit details
    Browse the repository at this point in the history

  5. enhance: Avoid assign too much segment/channels to new querynode (#34096 

    )

issue: #34095

When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.

This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    8123bea View commit details
    Browse the repository at this point in the history

  6. fix: LegacyVersionWithoutRPCWatch default value to 2.4.1 (#34184) 

    See also: #31933

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
    @XuanYang-cn
    XuanYang-cn committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    dda70aa View commit details
    Browse the repository at this point in the history

  7. enhance: [skip e2e] use disk ann when os is ubuntu (#34236) 

    issue: #34222

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
    @yellow-shine
    yellow-shine committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    b14482b View commit details
    Browse the repository at this point in the history

  8. fix: [GoSDK] Allocate correct bytes for bf16 vector (#34233) 

    See also #33848

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    97db7be View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2024

  1. test: remove dup code (#34242) 

    Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
    @zhuwenxing
    zhuwenxing committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    00e5a40 View commit details
    Browse the repository at this point in the history

  2. fix:load major compaction partial result(#34051) (#34052) 

    related: #34051

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
    @MrPresent-Han
    MrPresent-Han and MrPresent-Han committed Jun 28, 2024
    Configuration menu
    Copy the full SHA
    96dcee5 View commit details
    Browse the repository at this point in the history

Commits on Jun 30, 2024

  1. enhance: add log to bitmap index (#34197) 

    #32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
    @zhagnlu
    zhagnlu and luzhang committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    cc1bc07 View commit details
    Browse the repository at this point in the history

  2. enhance: refactor variable column to reduce memory cost (#33875) 

    #33874

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
    @zhagnlu
    zhagnlu and luzhang committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    3030e46 View commit details
    Browse the repository at this point in the history

  3. Refine clustering_compaction_task retry mechanism (#34194) 

    #32939

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    73ffc1b View commit details
    Browse the repository at this point in the history

  4. Use new stream segment reader in clustering compaction (#34232) 

    #32939

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jun 30, 2024
    Configuration menu
    Copy the full SHA
    e5d691d View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. enhance: improve check health (#33800) 

    issue: #34264

Signed-off-by: jaime <yun.zhang@zilliz.com>
    @jaime0815
    jaime0815 committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    0426390 View commit details
    Browse the repository at this point in the history

  2. enhance: Skip update index for L0 segment (#34099) 

    try to update index for l0 segment, will failed by `index not found`

This PR skip update index for l0 segment

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    f7ecafe View commit details
    Browse the repository at this point in the history

  3. fix: Implement singleflight for segcore ChunkCache (#34250) 

    See also #34249

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    14e827d View commit details
    Browse the repository at this point in the history

  4. enhance: make subfunctions of datanode component modular (#33992) 

    issue: #33994

also remove deprecated channel manager based on the etcd implementation

Signed-off-by: jaime <yun.zhang@zilliz.com>
    @jaime0815
    jaime0815 committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    d6afb31 View commit details
    Browse the repository at this point in the history

  5. enhance: support mark error as user error (#33498) 

    relate: #33492

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
    @aoiasd
    aoiasd committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    186757e View commit details
    Browse the repository at this point in the history

  6. enhance: Optimize grow slice cost during query (#34253) 

    issue: #32252 

This PR try to pre-allocate FieldData for Reduce operations in the Query
chain using typeutil.PrepareResultFieldData to avoid the overhead of
dynamically growing the slice during appendFieldData process.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    b49862d View commit details
    Browse the repository at this point in the history

  7. fix: Fix panic due to failed to seek (#34229) 

    Converting the same msgposition's vchannel to a pchannel multiple times
would result in an invalid pchannel, leading to seek failure and panic.
This PR:
1. Make a copy of msgposition in msgdispatcher.
2. Check if channel is already a pchannel, no further channel conversion
is performed.

issue: #34221

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
    @bigsheeper
    bigsheeper committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    ff51c7e View commit details
    Browse the repository at this point in the history

  8. fix: descriptor event in previous version not has nullable to parse e… 

    …rror (#34235)

#34176

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
    @smellthemoon
    smellthemoon and lixinguo committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    ef3ced8 View commit details
    Browse the repository at this point in the history

  9. fix: Check partition in current target when observing partition load … 

    …status (#34282)

See also #34234

`LoadPartitions` does not guarantee the current target has loading
partitions if there are some partitions already loaded before.

This PR check current target contains the partition to load when
advancing loading percentage to 100.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    b284b81 View commit details
    Browse the repository at this point in the history

  10. fix: Reduce duplicate PKs in segcore (#34267) 

    issue: #34247

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
    @bigsheeper
    bigsheeper committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    734415b View commit details
    Browse the repository at this point in the history

  11. enhance: Use channel name map finding channel watcher (#34294) 

    See also #33235

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    e083d99 View commit details
    Browse the repository at this point in the history

  12. enhance: Refine max length exceeded error message (#34300) 

    This PR make varchar & string array field max length exceeded error
message clearer. Also fixed a minor issue that error string format and
argument number not match.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    b32dae1 View commit details
    Browse the repository at this point in the history

  13. fix: Sync the sealed and flushed segments to datanode (#34301) 

    issue: #33696

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
    @xiaocai2333
    xiaocai2333 committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    4cf1a35 View commit details
    Browse the repository at this point in the history

  14. enhance: [GoSDK] Sync IndexType supported from v1 (#34310) 

    See also #31293

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 1, 2024
    Configuration menu
    Copy the full SHA
    87bccb1 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. enhance: Don't set the timeout for SyncSegments (#34226) 

    issue: #32809

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
    @xiaocai2333
    xiaocai2333 committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    a5be322 View commit details
    Browse the repository at this point in the history

  2. enhance: timetick interceptor implementation (#34238) 

    issue: #33285

- optimize the message package
- add interceptor package to achieve append operation intercepting.
- add timetick interceptor to attach timetick properties for message.
- add timetick background task to send timetick message.

Signed-off-by: chyezh <chyezh@outlook.com>
    @chyezh
    chyezh committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    3563136 View commit details
    Browse the repository at this point in the history

  3. enhance: update knowhere version (#34361) 

    Signed-off-by: chasingegg <chao.gao@zilliz.com>
    @chasingegg
    chasingegg committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    ec5db7a View commit details
    Browse the repository at this point in the history

  4. fix: update check for sparse hnsw index (#33713) 

    issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
    @zhengbuqian
    zhengbuqian committed Jul 2, 2024
    Configuration menu
    Copy the full SHA
    fa8d641 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. enhance: add metrics for segment prune latnecy(#30376) (#34094) 

    related: #30376

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
    @MrPresent-Han
    MrPresent-Han and MrPresent-Han committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    34bec2e View commit details
    Browse the repository at this point in the history

  2. fix: Revert a optimize in clustering compaction (#34299) 

    #30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    fb61344 View commit details
    Browse the repository at this point in the history

  3. enhance: [skip e2e] update the version of MV (#34378) 

    issue: #29892

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
    @PwzXxm
    PwzXxm committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    32c943f View commit details
    Browse the repository at this point in the history

  4. enhance: Tag gotestsum version when install deps (#34308) 

    Tagging gotestsum by ldflags to prevent reinstall gotestsum binary each
local run

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    776ffee View commit details
    Browse the repository at this point in the history

  5. enhance: Optimeize clustering compaction (#34313) 

    issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
    @xiaocai2333
    xiaocai2333 committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    8165961 View commit details
    Browse the repository at this point in the history

  6. enhance: Skip pick worker when task doesn't need to execute actually (#… 

    …34348)

issue: #34347

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
    @xiaocai2333
    xiaocai2333 committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    feb13cd View commit details
    Browse the repository at this point in the history

  7. fix: Use raw parameter value to perform CAS (#34343) 

    See also #34342

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    94fb580 View commit details
    Browse the repository at this point in the history

  8. fix: Only load or release Flushed segment in datanode meta (#34390) 

    issue: #34376 , #34379 , #34375

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
    @xiaocai2333
    xiaocai2333 committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    2e434e4 View commit details
    Browse the repository at this point in the history

  9. fix: remove isFull check in compaction.enqueue (#34338) 

    #30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    a501fa1 View commit details
    Browse the repository at this point in the history

  10. doc: [skip e2e] add extraConfig to embed milvus (#34395) 

    Signed-off-by: ping.liu <ping.liu@zilliz.com>
    @pingliu
    pingliu committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    8c42f13 View commit details
    Browse the repository at this point in the history

  11. fix: Avoid datarace in clustering compaction (#34288) 

    #34289

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    da56880 View commit details
    Browse the repository at this point in the history

  12. enhance: Fix lint issues & sdk testcase (#34399) 

    Some lint issue is not detect due to recent static check pipeline issue.
This PR fixes these problem and Go milvusclient testcases.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    0fd0fcf View commit details
    Browse the repository at this point in the history

  13. enhance: remove deprecated code within channel manager (#34340) 

    issue: #33994

only remove deprecated code, no additional changes.

Signed-off-by: jaime <yun.zhang@zilliz.com>
    @jaime0815
    jaime0815 committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    d1f57aa View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2024

  1. fix: Remove unused code to fix compiling (#34406) 

    issue: #34407

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
    @xiaocai2333
    xiaocai2333 committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    23e2a0e View commit details
    Browse the repository at this point in the history

  2. enhance: refine misleading param name for bloom filter parallel factor ( 

    #34334)

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    9db8979 View commit details
    Browse the repository at this point in the history

  3. fix: DataNode might OOM by estimating based on MemorySize (#34201) 

    See also: #34136

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
    @XuanYang-cn
    XuanYang-cn committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    8a2be8a View commit details
    Browse the repository at this point in the history

  4. enhance: add the tick log for the tt msgstream seek method (#34397) 

    /kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
    @SimFG
    SimFG committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    c6e2900 View commit details
    Browse the repository at this point in the history

  5. enhance: Continue loop when reassign channel fails (#34331) 

    Log will be confusing when `Reassign` channel operation failed for both
success & failure log will be printed in row. This PR continue the loop
to avoid this output.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    d51d095 View commit details
    Browse the repository at this point in the history

  6. enhance: use configed max topk for iterator when input topk exceeds (#… 

    …34290)

related: #34290

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
    @MrPresent-Han
    MrPresent-Han and MrPresent-Han committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    10b3ce2 View commit details
    Browse the repository at this point in the history

  7. test: add cases for gosdk v2 upsert (#34400) 

    - test: add cases for gosdk v2 upsert
- test: update clientv2 to client reference

issue: #33419

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
    @ThreadDao
    ThreadDao committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    e4cece8 View commit details
    Browse the repository at this point in the history

  8. enhance: wal adaptor implementation (#34122) 

    issue: #33285

- add adaptor to implement walimpls into wal interface.
- implement timetick sorted and filtering scanner.
- add test for wal.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
    @chyezh
    chyezh committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    7611128 View commit details
    Browse the repository at this point in the history

  9. fix: Correct the update logic of timerecorder (#34339) 

    Correct the update logic of timerecorder in the flowgraph to avoid false
failure: "some node(s) haven't received input".

issue: #34337

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
    @bigsheeper
    bigsheeper committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    0b404bf View commit details
    Browse the repository at this point in the history

  10. enhance: reconstruct scalar part's code for segment-pruner(#30376) (#… 

    …34346)

related: #30376
1. support more complex expr
2. add more ut test for unrelated fields

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
    @MrPresent-Han
    MrPresent-Han and MrPresent-Han committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    fcafdb6 View commit details
    Browse the repository at this point in the history

  11. fix: Skip l0 segments when syncing segments to datanodes (#34388) 

    See also #34387

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    439fb24 View commit details
    Browse the repository at this point in the history

  12. fix: [restful v2] count(*) & hook (#34369) 

    issue:  #31224 #34374

for query api:
1. param filter is not requried
2. param limit is useless while outputFields = [count(*)]

add hook about grpc call

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
    @PowderLi
    PowderLi committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    c16a704 View commit details
    Browse the repository at this point in the history

  13. [automated] Update Knowhere Commit (#34432) 

    Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
    @sre-ci-robot @github-actions
    sre-ci-robot and github-actions[bot] committed Jul 4, 2024
    Configuration menu
    Copy the full SHA
    d65b689 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2024

  1. test: add test cases for gosdk v2 index (#34431) 

    issue: #33419 
- Add test cases for gosdk v2 index
- Add sparse index for go client

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
    @ThreadDao
    ThreadDao committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    6774724 View commit details
    Browse the repository at this point in the history

  2. enhance: implement wal managerment on streaming node (#34153) 

    issue: #33285

- add lifetime control for wal.
- implement distributed-safe wal manager on streaming node.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
    @chyezh
    chyezh committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    ba04981 View commit details
    Browse the repository at this point in the history

  3. doc: Update # of GitHub stars in README.md (#33577) 

    The # of stars for https://github.com/milvus-io/milvus/ is over 27k now,
though https://zilliz.com/what-is-milvus links to
https://github.com/zilliztech, and [the fork
there](https://github.com/zilliztech/milvus) has only 23 stars.
    @dandv
    dandv committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    7cac20f View commit details
    Browse the repository at this point in the history

  4. enhance: use the key lock and concurrent map in the msg dispatcher cl… 

    …ient (#34278)

/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
    @SimFG
    SimFG committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    0817802 View commit details
    Browse the repository at this point in the history

  5. enhance: Remove datanode reporting TT based on MQ implementation (#34421 

    )

issue: #34420

Signed-off-by: jaime <yun.zhang@zilliz.com>
    @jaime0815
    jaime0815 committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    21fc5f5 View commit details
    Browse the repository at this point in the history

  6. fix: fix bitmap supported type (#34350) 

    #34314

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
    @zhagnlu
    zhagnlu and luzhang committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    18f2458 View commit details
    Browse the repository at this point in the history

  7. fix: generate gin logs failed cause access logs panic (#34142) 

    relate: #34141

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
    @aoiasd
    aoiasd committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    233b848 View commit details
    Browse the repository at this point in the history

  8. fix: Write padding into mmap file in case of SIGBUS (#34443) 

    See also #34442

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    6b4d977 View commit details
    Browse the repository at this point in the history

  9. fix: [GoSDK] Avoid creating schema for non-fast CreateCollection (#34448 

    )

See also #31293

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    a1c65de View commit details
    Browse the repository at this point in the history

  10. enhance: add disk quota and max collections into db properties (#34368) 

    issue: #34385

Signed-off-by: jaime <yun.zhang@zilliz.com>
    @jaime0815
    jaime0815 committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    60be454 View commit details
    Browse the repository at this point in the history

  11. enhance: retry etcd server start with 3 default retries (#34208) 

    Added retry method and unit test cases for retrying etcd server start.
New to open source and Go language. Please reject if this is not the
requirement/ specify the modifications needed in code.
issue : #17569

Signed-off-by: Charles Kakumanu <charles_kakumanu@apple.com>
Co-authored-by: Charles Kakumanu <charles_kakumanu@apple.com>
    @charleskakumanu
    charleskakumanu and Charles Kakumanu committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    05df709 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2024

  1. enhance: Check segment existence when FlushSegments and add some key … 

    …logs (#34438)

Check if the segment exists during FlushSegments and add some key logs
in write path.

issue: #34255

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
    @bigsheeper
    bigsheeper committed Jul 6, 2024
    Configuration menu
    Copy the full SHA
    43fd8d1 View commit details
    Browse the repository at this point in the history

  2. enhance: add delta log stream new format reader and writer (#34116) 

    issue: #34123

Benchmark case: The benchmark run the go benchmark function
`BenchmarkDeltalogFormat` which is put in the Files changed. It tests
the performance of serializing and deserializing from two different data
formats under a 10 million delete log dataset.

Metrics: The benchmarks measure the average time taken per operation
(ns/op), memory allocated per operation (MB/op), and the number of
memory allocations per operation (allocs/op).
| Test Name | Avg Time (ns/op) | Time Comparison | Memory Allocation
(MB/op) | Memory Comparison | Allocation Count (allocs/op) | Allocation
Comparison |

|---------------------------------|------------------|-----------------|---------------------------|-------------------|------------------------------|------------------------|
| one_string_format_reader | 2,781,990,000 | Baseline | 2,422 | Baseline
| 20,336,539 | Baseline |
| pk_ts_separate_format_reader | 480,682,639 | -82.72% | 1,765 | -27.14%
| 20,396,958 | +0.30% |
| one_string_format_writer | 5,483,436,041 | Baseline | 13,900 |
Baseline | 70,057,473 | Baseline |
| pk_and_ts_separate_format_writer| 798,591,584 | -85.43% | 2,178 |
-84.34% | 30,270,488 | -56.78% |

Both read and write operations show significant improvements in both
speed and memory allocation.

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
    @shaoting-huang
    shaoting-huang committed Jul 6, 2024
    Configuration menu
    Copy the full SHA
    f4dd7c7 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2024

  1. enhance: Pre-allocate ids for import (#33958) 

    The import is dependent on syncTask, which in turn relies on the
allocator. This PR pre-allocate the necessary IDs for import syncTask.

issue: #33957

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
    @bigsheeper
    bigsheeper committed Jul 7, 2024
    Configuration menu
    Copy the full SHA
    4e5f1d5 View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2024

  1. fix: fix clustering compaction can't enqueue when compaction queue is… 

    … full (#34445)

#30633

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    efdaed4 View commit details
    Browse the repository at this point in the history

  2. fix: Add nbits parameter check for IVF_PQ (#34451) 

    See also #34426

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    145f4f9 View commit details
    Browse the repository at this point in the history

  3. test: update test case (#34108) 

    Signed-off-by: elstic <hao.wang@zilliz.com>
    @elstic
    elstic committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    f7898a3 View commit details
    Browse the repository at this point in the history

  4. fix: Accidently exit the check loop (#34481) 

    See also: #34460

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
    @XuanYang-cn
    XuanYang-cn committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    467113d View commit details
    Browse the repository at this point in the history

  5. fix: Upgraded Azidentity Package to v1.6.0 (#34464) 

    issue : #34456

Signed-off-by: Ald392 <imagesai32@gmail.com>
    @Ald392
    Ald392 committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    686a212 View commit details
    Browse the repository at this point in the history

  6. enhance: change autoindex default metric type (#34261) 

    issue: #34304 
cosine is more widely used in float vectors, and cosine and hamming
distance are 'metrics' which have good geometric properties

Signed-off-by: chasingegg <chao.gao@zilliz.com>
    @chasingegg
    chasingegg committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    ae6d6f9 View commit details
    Browse the repository at this point in the history

  7. fix: lose partitionIDs when scalar pruning and refine segment prune r… 

    …atio metrics(#30376) (#34477)

related: #30376
fix: paritionIDs lost when no setting paritions
enhance: refine metrics for segment prune

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
    @MrPresent-Han
    MrPresent-Han committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    8af187f View commit details
    Browse the repository at this point in the history

  8. fix: Fix incorrect segment num rows (#34441) 

    Repeated calls to UpdateStatistics, this PR correct it.

issue: #34440

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
    @bigsheeper
    bigsheeper committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    e15ac2b View commit details
    Browse the repository at this point in the history

  9. fix: Check err is ErrKeyNotFound when CASCachedValue (#34488) 

    See also #33785

When config item is not present in paramtable, CAS fails due to
GetConfig returns error.

This PR make this returned err instance of ErrKeyNotFound and check
error type in \`CASCachedValue\` methods.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    80b620e View commit details
    Browse the repository at this point in the history

  10. enhance: implement streaming node server service (#34166) 

    issue: #33285

- implement producing and consuming server of message
- implement management operation for streaming node server

---------

Signed-off-by: chyezh <chyezh@outlook.com>
    @chyezh
    chyezh committed Jul 8, 2024
    Configuration menu
    Copy the full SHA
    dfe0416 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2024

  1. enhance: update milvus proto (#34490) 

    issue: #34336

Revert CollectionSchema changes

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
    @PwzXxm
    PwzXxm committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    df5ba3f View commit details
    Browse the repository at this point in the history

  2. enhance: Fix lint issues from recent PRs (#34482) 

    See also #34483
Some lint issues are introduced due to lack of static check run. This PR
fixes these problems.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    3333160 View commit details
    Browse the repository at this point in the history

  3. fix: Fix unstable ut of msgdispatcher (#34485) 

    issue: #34484

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
    @bigsheeper
    bigsheeper committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    391d2aa View commit details
    Browse the repository at this point in the history

  4. fix: fix compaction task startTime is 0 (#34496) 

    fix bug: #34497 #34498

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    0c4d82c View commit details
    Browse the repository at this point in the history

  5. fix: Query may return deleted records (#34501) 

    issue: #34500
cause the sort in `GetLevel0Deletions` will broken the corresponed order
between pks and tss, then the pks and tss will be sorted in
segment.Delete() interface.

This PR remove this uncessary and incorrect sort progress to avoid query
may return deleted records.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    eeb03a0 View commit details
    Browse the repository at this point in the history

  6. fix: reuse deserialize result to help improve memory management (#34507) 

    Fixed #33268
The original reuse is broken by #33359

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
    @tedxu
    tedxu committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    eae4dfc View commit details
    Browse the repository at this point in the history

  7. enhance: release the record in delete codec and add some log for comp… 

    …action (#34454)

/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
    @SimFG
    SimFG committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    5016038 View commit details
    Browse the repository at this point in the history

  8. enhance: support setting properties in create database request (#34510) 

    issue: #34493

Signed-off-by: jaime <yun.zhang@zilliz.com>
    @jaime0815
    jaime0815 committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    4365308 View commit details
    Browse the repository at this point in the history

  9. enhance:change access log write cache default config (#34354) 

    Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
    @aoiasd
    aoiasd committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    c1e0453 View commit details
    Browse the repository at this point in the history

  10. enhance: Tidy dc channel manager (#34515) 

    See also: #34518

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
    @XuanYang-cn
    XuanYang-cn committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    314f4d9 View commit details
    Browse the repository at this point in the history

  11. [automated] Update Knowhere Commit (#34549) 

    Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
    @sre-ci-robot @github-actions
    sre-ci-robot and github-actions[bot] committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    0b3f317 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2024

  1. Update all contributors 

    Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
    @sre-ci-robot
    sre-ci-robot committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    2f70bbe View commit details
    Browse the repository at this point in the history

  2. fix: RootCoord panic during check health (#34541) 

    issue: #34540

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    3c70463 View commit details
    Browse the repository at this point in the history

  3. enhance: Avoid use concrete segment type in segments interfaces (#34521) 

    See also #34519

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    d60e628 View commit details
    Browse the repository at this point in the history

  4. enhance: Enable setting the replica number and resource group during … 

    …collection creation (#34403)

issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    9b37d3f View commit details
    Browse the repository at this point in the history

  5. fix: Write padding at end of mmap file not chunk (#34529) 

    Related to #34508

The padding bytes shall be written only at the end of the mmap file not
the chunk of each field data file.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    4850336 View commit details
    Browse the repository at this point in the history

  6. fix: Indexnode stuck at stopping progress cause by wrong lifetime con… 

    …trol (#34558)

issue: #34557

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
    @weiliu1031
    weiliu1031 committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    ebc68d2 View commit details
    Browse the repository at this point in the history

  7. enhance: add the seal segment when dispatch delete msgs (#34565) 

    /kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
    @SimFG
    SimFG committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    b58a561 View commit details
    Browse the repository at this point in the history

  8. fix: Prevent dispatcher merging if curTs is 0 (#34562) 

    When the main dispatcher has not yet consumed data, curTs is 0. During
this time, merging dispatchers should not be allowed; otherwise, the
data of the solo dispatcher will be skipped.

issue: #34255

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
    @bigsheeper
    bigsheeper committed Jul 10, 2024
    Configuration menu
    Copy the full SHA
    aef7664 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. enhance: skip orphan channel cp meta when checking cp lag (#34555) 

    issue: # #34545

Print warn log instead of check health fail if orphan channel cp meta is
found in health check request.

Signed-off-by: jaime <yun.zhang@zilliz.com>
    @jaime0815
    jaime0815 committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    c332f69 View commit details
    Browse the repository at this point in the history

  2. enhance: implement balancer at streaming coord (#34435) 

    issue: #33285

- add balancer implementation
- add channel count fair balance policy
- add channel assignment discover grpc service

Signed-off-by: chyezh <chyezh@outlook.com>
    @chyezh
    chyezh committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    1bc3c0b View commit details
    Browse the repository at this point in the history

  3. fix: unstable UT for level0 deletion (#34524) 

    issue: #34533

Signed-off-by: jaime <yun.zhang@zilliz.com>
    @jaime0815
    jaime0815 committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    3b62138 View commit details
    Browse the repository at this point in the history

  4. enhance: Print reason when seal segments (#34575) 

    See also #34574

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    be92147 View commit details
    Browse the repository at this point in the history

  5. fix: Milvus panic when compaction disabled and dropping a collection (#… 

    …34103)

See also: #31059

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
    @XuanYang-cn
    XuanYang-cn committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    e0b39d8 View commit details
    Browse the repository at this point in the history

  6. test: add cases for gosdk v2 partitions (#34593) 

    issue: #33419 #34426

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
    @ThreadDao
    ThreadDao committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    e411548 View commit details
    Browse the repository at this point in the history

  7. enhance: support upsert autoid==true (#30342) 

    related with: #29258

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
    @smellthemoon
    smellthemoon and lixinguo committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    07b94b4 View commit details
    Browse the repository at this point in the history

  8. enhance: Refine clustering compaction latency metrics (#34528) 

    #34527

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    fd3da90 View commit details
    Browse the repository at this point in the history

  9. enhance: Add back compactionTaskNum metrics (#34583) 

    Fix L0 compaction task recover unable to set segment not isCompacting

See also: #34460

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
    @XuanYang-cn
    XuanYang-cn committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    d7a3697 View commit details
    Browse the repository at this point in the history

  10. feat: support partition key isolation (#34336) 

    issue: #34332

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
    @PwzXxm
    PwzXxm committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    104d096 View commit details
    Browse the repository at this point in the history

  11. fix: Compaction task l0 state transfer wrong (#34597) 

    This bug caused failed L0 compaction tasks never end.

See also: #34460

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
    @XuanYang-cn
    XuanYang-cn committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    d7966f4 View commit details
    Browse the repository at this point in the history

  12. fix: encountering orphan channel-cp meta after DataCoord GC (#34612) 

    issue: #34545

Signed-off-by: jaime <yun.zhang@zilliz.com>
    @jaime0815
    jaime0815 committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    a08a0c8 View commit details
    Browse the repository at this point in the history

  13. enhance: Alter compactTo segments before compactFrom to avoid data lo… 

    …ss if crash (#34513)

#34512

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    358e9a1 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. fix: Restful API use deprecate error code cause access log panic. (#3… 

    …4576)

relate: #34578

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
    @aoiasd
    aoiasd committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    5bb0d21 View commit details
    Browse the repository at this point in the history

  2. feat: support group_size for search_group_by(#33544) (#33720) 

    related: #33544

mainly changes in three aspects:

1. enable setting group_size for group by function
2. separate normal reduce and group by reduce
3. eleminate uncessary padding in search result for reducing

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
    @MrPresent-Han
    MrPresent-Han and MrPresent-Han committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    f00c529 View commit details
    Browse the repository at this point in the history

  3. enhance: Add lint rule to forbid gogo protobuf (#34594) 

    github.com/gogo/protobuf is deprecated and could be error prune after
upgrade protobuf message to v2.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    531092c View commit details
    Browse the repository at this point in the history

  4. enhance: use the insert rate limit for the upsert request (#34615) 

    - issue: #34614

Signed-off-by: SimFG <bang.fu@zilliz.com>
    @SimFG
    SimFG committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    d8e68cb View commit details
    Browse the repository at this point in the history

  5. enhance: Add param item for segmentFlushInterval (#34629) 

    See also #28817

Add paramitem for segment flush interval

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
    @congqixia
    congqixia committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    1a248f2 View commit details
    Browse the repository at this point in the history

  6. fix: Remove flushed segment in segment manager generated through impo… 

    …rt (#34649)

issue: #34648

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
    @xiaocai2333
    xiaocai2333 committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    93a5006 View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2024

  1. test: Fix index creation mismatch for sparse vector columns in multiv… 

    …ector case (#34618)

If `is_all_data_type` is true, the case will add float32, f16, bf16, and
sparse vectors, but the created indexes are all `flat` indexes by
default. The sparse type cannot create a flat index. Fix the test code
to create a `SPARSE_INVERTED_INDEX` index for the sparse vector when
is_all_data_type is true

Signed-off-by: elstic <hao.wang@zilliz.com>
    @elstic
    elstic committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    2c462d3 View commit details
    Browse the repository at this point in the history

  2. fix: fix auto merge error (#34661) 

    #33704

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
    @zhagnlu
    zhagnlu and luzhang committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    18c83c6 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2024

  1. fix: fix bug that set incorrect info to columnbase (#34428) 

    #34427

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
    @zhagnlu
    zhagnlu and luzhang committed Jul 14, 2024
    Configuration menu
    Copy the full SHA
    bd9727a View commit details
    Browse the repository at this point in the history

  2. fix: fix meta revert bug when clustering compaction task fail (#34604) 

    #34603

Signed-off-by: wayblink <anyang.wang@zilliz.com>
    @wayblink
    wayblink committed Jul 14, 2024
    Configuration menu
    Copy the full SHA
    48898ce View commit details
    Browse the repository at this point in the history