Skip to content

[fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking.#63666

Open
kaka11chen wants to merge 1 commit into
apache:masterfrom
kaka11chen:fix_shared_vexpr_in_ann_range_search
Open

[fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking.#63666
kaka11chen wants to merge 1 commit into
apache:masterfrom
kaka11chen:fix_shared_vexpr_in_ann_range_search

Conversation

@kaka11chen
Copy link
Copy Markdown
Contributor

@kaka11chen kaka11chen commented May 26, 2026

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

ANN range search execution state was stored on shared VExpr roots. VExprContext clones share the root expression, so a segment that executed ANN range search could leak that state into another segment without an ANN index and incorrectly remove the common expression. ANN range search also mixed schema column indexes with storage column ids when updating common expression index status, so remapped schemas failed to mark the source slot expression as evaluated. This patch returns ANN execution state through the current evaluation call, stores ANN root bitmap in the current segment IndexContext, and updates slot index status by source column index.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@kaka11chen
[fix](be) Track ANN range search per segment
ac75639 
Issue Number: None

Related PR: None

Problem Summary: ANN range search execution state was stored on shared VExpr roots. VExprContext clones share the root expression, so a segment that executed ANN range search could leak that state into another segment without an ANN index and incorrectly remove the common expression. ANN range search also mixed schema column indexes with storage column ids when updating common expression index status, so remapped schemas failed to mark the source slot expression as evaluated. This patch returns ANN execution state through the current evaluation call, stores ANN root bitmap in the current segment IndexContext, and updates slot index status by source column index.

None

- Test: Manual test
    - git diff --check
    - build-support/check-format.sh attempted but blocked because the environment has clang-format 20.1.8 while Doris requires clang-format 16
    - BE UT not run to avoid starting a compile after the compile stop request
- Behavior changed: No
- Does this need documentation: No
@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 26, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen
Copy link
Copy Markdown
Contributor Author

kaka11chen commented May 26, 2026

run buildall

@kaka11chen kaka11chen changed the title [fix](ann-index) Fix ANN Range Search State Leakage and Incorrect Slot Index Tracking. [fix](ann-index) Fix ANN range search state leakage and incorrect slot index tracking. May 26, 2026
@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 26, 2026

TPC-H: Total hot run time: 32234 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ac75639cd1ea6888ce54d1c6cbefc99d6b360d3c, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17854	4149	4070	4070
q2	q3	10833	1457	831	831
q4	4684	488	354	354
q5	7549	2345	2171	2171
q6	236	183	141	141
q7	992	787	649	649
q8	9456	1800	1670	1670
q9	5373	5060	5025	5025
q10	6431	2250	1918	1918
q11	448	267	247	247
q12	637	443	301	301
q13	18184	3410	2797	2797
q14	265	264	236	236
q15	q16	826	776	718	718
q17	965	1065	928	928
q18	7119	5709	5575	5575
q19	1292	1268	1256	1256
q20	550	421	286	286
q21	6293	2967	2750	2750
q22	437	375	311	311
Total cold run time: 100424 ms
Total hot run time: 32234 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4893	4845	4873	4845
q2	q3	4934	5270	4708	4708
q4	2211	2255	1448	1448
q5	4923	4723	4752	4723
q6	233	182	137	137
q7	1875	1763	1601	1601
q8	2521	2179	2279	2179
q9	7526	7471	7522	7471
q10	4807	4749	4276	4276
q11	535	387	366	366
q12	743	758	598	598
q13	3026	3404	2849	2849
q14	272	277	258	258
q15	q16	693	708	624	624
q17	1322	1289	1283	1283
q18	7345	7070	6941	6941
q19	1116	1101	1108	1101
q20	2236	2241	1980	1980
q21	5383	4700	4592	4592
q22	564	483	412	412
Total cold run time: 57158 ms
Total hot run time: 52392 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 26, 2026

TPC-DS: Total hot run time: 172815 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ac75639cd1ea6888ce54d1c6cbefc99d6b360d3c, data reload: false

query5	4320	649	521	521
query6	335	215	214	214
query7	4233	585	306	306
query8	328	233	217	217
query9	8882	4164	4159	4159
query10	451	353	333	333
query11	5826	2587	2329	2329
query12	177	130	124	124
query13	1334	560	437	437
query14	6059	5485	5208	5208
query14_1	4514	4499	4496	4496
query15	212	202	183	183
query16	991	433	418	418
query17	936	708	586	586
query18	2471	506	366	366
query19	217	214	168	168
query20	138	134	140	134
query21	215	140	127	127
query22	13630	13616	13455	13455
query23	17444	16590	16231	16231
query23_1	16501	16347	16394	16347
query24	7447	1809	1336	1336
query24_1	1362	1341	1327	1327
query25	595	509	437	437
query26	1313	319	181	181
query27	2683	594	355	355
query28	4502	2030	2037	2030
query29	997	650	524	524
query30	294	247	201	201
query31	1148	1089	971	971
query32	85	80	76	76
query33	540	368	308	308
query34	1201	1137	667	667
query35	787	791	703	703
query36	1449	1394	1300	1300
query37	158	106	94	94
query38	3240	3188	3070	3070
query39	940	915	895	895
query39_1	879	879	873	873
query40	240	161	130	130
query41	73	69	71	69
query42	119	113	113	113
query43	332	340	309	309
query44	
query45	215	209	202	202
query46	1105	1202	756	756
query47	2440	2413	2269	2269
query48	398	401	309	309
query49	651	512	407	407
query50	945	360	264	264
query51	4374	4325	4254	4254
query52	108	107	99	99
query53	262	285	210	210
query54	342	284	265	265
query55	96	93	89	89
query56	301	313	307	307
query57	1436	1441	1338	1338
query58	346	264	261	261
query59	1580	1741	1431	1431
query60	320	312	306	306
query61	159	155	148	148
query62	698	651	586	586
query63	236	199	204	199
query64	2392	796	611	611
query65	
query66	1747	480	350	350
query67	29865	29899	29666	29666
query68	
query69	464	349	305	305
query70	1075	998	1011	998
query71	308	276	263	263
query72	3065	2732	2448	2448
query73	899	761	426	426
query74	5148	4961	4807	4807
query75	2699	2609	2253	2253
query76	2313	1155	801	801
query77	420	400	370	370
query78	12535	12677	11969	11969
query79	1321	1098	779	779
query80	600	537	458	458
query81	464	286	240	240
query82	242	154	122	122
query83	380	283	243	243
query84	306	140	114	114
query85	966	554	458	458
query86	414	349	313	313
query87	3443	3392	3244	3244
query88	3620	2758	2745	2745
query89	429	406	339	339
query90	1837	186	179	179
query91	183	166	139	139
query92	79	79	73	73
query93	1464	1433	846	846
query94	554	358	296	296
query95	656	391	434	391
query96	1113	811	357	357
query97	2719	2723	2602	2602
query98	235	236	227	227
query99	1175	1142	1044	1044
Total cold run time: 253752 ms
Total hot run time: 172815 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 26, 2026

BE Regression && UT Coverage Report

Increment line coverage 95.74% (45/47) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.75% (28037/38018)
Line Coverage 57.64% (304228/527814)
Region Coverage 54.79% (254543/464592)
Branch Coverage 56.27% (109909/195333)

@kaka11chen kaka11chen marked this pull request as ready for review May 26, 2026 11:18
@kaka11chen kaka11chen requested a review from yiguolei as a code owner May 26, 2026 11:18
Copilot AI review requested due to automatic review settings May 26, 2026 11:18
Copilot AI reviewed May 26, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@yiguolei yiguolei Awaiting requested review from yiguolei yiguolei is a code owner

@liaoxin01 liaoxin01 Awaiting requested review from liaoxin01 liaoxin01 is a code owner

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kaka11chen @hello-stephen