Skip to content

Merge master vec #51109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
zhiqiang-hhhh wants to merge 23 commits into from
Closed

Merge master vec #51109

zhiqiang-hhhh wants to merge 23 commits into from

Conversation

@zhiqiang-hhhh
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh commented May 21, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

zhiqiang-hhhh and others added 22 commits April 1, 2025 15:53
@zhiqiang-hhhh @chenlinzhong
[vector] fix compile on veector-index branch (apache#49733)
6d26690 
rebase apache#49703 on master
rm diskann src code (impl is kept as reference)
fix BE comple
fix fmt

NOTE: compilation of FE still has error.

---------

Co-authored-by: chenlinzhong <490103404@qq.com>
@zhiqiang-hhhh
enable faiss hnsw (apache#49745)
2a915e4 
```
CREATE TABLE `vector_table` (
  `siteid` int(11) NULL DEFAULT "10" COMMENT "",
  `embedding` array<float>  NOT NULL  COMMENT "",
  `comment` text NULL,
  INDEX idx_test_ann (`embedding`) USING ANN PROPERTIES(
    "index_type"="hnsw",
    "metric_type"="l2",
    "dim"="8",
    "max_degree"="100") COMMENT 'test diskann index',
  INDEX idx_comment (`comment`) USING INVERTED PROPERTIES("support_phrase" = "true", "parser" = "english", "lower_case" = "true") COMMENT 'inverted index for comment' )
  ENGINE=OLAP duplicate KEY(`siteid`) COMMENT "OLAP" DISTRIBUTED BY HASH(`siteid`) BUCKETS 1 PROPERTIES ( "replication_num" = "1" );

INSERT INTO `vector_table` (`siteid`, `embedding`,`comment`) VALUES
(10, [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0,20],"emb1"),
(20, [7.0, 6.0, 5.0, 4.0, 3.0, 2.0, 1.0,30],"emb2")
--------------

Query OK, 2 rows affected (0.07 sec)
{'label':'label_858347013b14baf_b9db5d59b5e30322', 'status':'VISIBLE', 'txnId':'18029'}
```

```
I20250401 19:18:17.977408 3765348 faiss_vector_index.cpp:86] Faiss index saved to faiss.idx, rows 2
```
@zzzxl1993
[fetura](vector) Extend the index interface to support vector indexin…
b2da241 
…g. (apache#49780)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@morrySnow
[feature](index) update phycical plan def for vector index
003f8e6
@zzzxl1993
[fix](inverted index) Fix compilation error (apache#50286)
f534ac5
@zhiqiang-hhhh
add is_virtual_slot (apache#50390)
24ff0fe
@morrySnow
[opt](Nereids) push down virtual column into scan (apache#50521)
f5491c9
@morrySnow
[fix](vector) column prune is wrong on virtual column (apache#50550)
8fea358
@morrySnow
[fix](vector) equals and hashcode are wrong in olap relation (apache#…
c3e0a10 
…50558)
@morrySnow
[fix](vector) should retain all virtual columns (apache#50571)
82b7c5d
@morrySnow
[feature](vector) push down ann topn into scan
e2c2881
@morrySnow
[opt](vector) reopen push down virtual column in filter rule
211df14
@zhiqiang-hhhh
remove all x_ prefix (apache#50995)
90d1b3d 
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@morrySnow
[opt](vector) refactor some scan code
6798572
@zhiqiang-hhhh
[feat] VirtualColumn & ANNTopN & AnnRangeSearch (apache#50403)
0be7da8 
This is the first version with relatively stable behavior:

* Pass the TPC-H test without crashes.

* Perform range search without crashes.

* Perform ANN Top-N search without crashes.

* Perform compound search (range + Top-N) without crashes.

Some unit tests are unstable; they may be related to the Faiss library.

```text
[  FAILED  ] 4 tests, listed below:
[  FAILED  ] VectorSearchTest.AnnTopNDescriptorEvaluateTopN
[  FAILED  ] VectorSearchTest.CompRangeSearch
[  FAILED  ] VectorSearchTest.RangeSearchNoSelector1
[  FAILED  ] VectorSearchTest.RangeSearchWithSelector1
```

tpch test has some unstable failure:
```text
q13	Error: Failed to execute query q13 (cold run). Output:
ERROR 1105 (HY000) at line 18: errCode = 2, detailMessage = (10.16.10.2)[INTERNAL_ERROR]Parameters start = 0, length = 4064, are out of bound in ColumnVector<T>::insert_range_from method (data.size() = 0).
```
@zhiqiang-hhhh
fix virtual column expr context (apache#51085)
756d64b 
failure of tpchq13 is fixed.
@morrySnow
[opt](vector) only extract l2_distance from filter
9421e5a
@zhiqiang-hhhh
Merge branch 'master' of github.com:apache/doris into vector-index-dev
341f1df
@zhiqiang-hhhh
fix merge be
44a93c0
@morrySnow
[opt](vector) only extract l2_distance from filter
3d56e7e
@zhiqiang-hhhh
Merge branch 'vector-index-dev' of github.com:apache/doris into merge…
8d051bd 
…-master-vec
@zhiqiang-hhhh
fix ut
defe918
@hello-stephen
Copy link
Contributor

hello-stephen commented May 21, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zclllyybb zclllyybb Awaiting requested review from zclllyybb zclllyybb will be requested when the pull request is marked ready for review zclllyybb is a code owner

@BiteTheDDDDt BiteTheDDDDt Awaiting requested review from BiteTheDDDDt BiteTheDDDDt will be requested when the pull request is marked ready for review BiteTheDDDDt is a code owner

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou will be requested when the pull request is marked ready for review gavinchou is a code owner

@dataroaring dataroaring Awaiting requested review from dataroaring dataroaring will be requested when the pull request is marked ready for review dataroaring is a code owner

@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei will be requested when the pull request is marked ready for review yiguolei is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zhiqiang-hhhh @hello-stephen @zzzxl1993 @morrySnow