Skip to content

[fix](index) remove OR predicate support for bitmap index#30951

Merged
qidaye merged 1 commit intoapache:masterfrom
xiaokang:diable_bitmap_index_or
Feb 7, 2024
Merged

[fix](index) remove OR predicate support for bitmap index#30951
qidaye merged 1 commit intoapache:masterfrom
xiaokang:diable_bitmap_index_or

Conversation

@xiaokang
Copy link
Contributor

@xiaokang xiaokang commented Feb 7, 2024 •
edited
Loading

Proposed changes

There is some problem for OR predicate push down using bitmap index, so disable it.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Feb 7, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@xiaokang
Copy link
Contributor Author

xiaokang commented Feb 7, 2024

run buildall

@xiaokang xiaokang changed the title [feature](index) disable OR predicate for bitmap index [feature](index) remove OR predicate support for bitmap index Feb 7, 2024
@xiaokang xiaokang added usercase Important user case type label dev/2.0.5 labels Feb 7, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Feb 7, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Feb 7, 2024

TPC-H: Total hot run time: 37090 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d28afd1b3c8e80f75a0120d50812084bb90a5d84, data reload: false

------ Round 1 ----------------------------------
q1	17601	4488	4464	4464
q2	2042	140	130	130
q3	10925	945	920	920
q4	4650	788	732	732
q5	7690	2872	2829	2829
q6	184	121	119	119
q7	1141	727	723	723
q8	9286	2005	2022	2005
q9	7258	6355	6369	6355
q10	8108	2424	2389	2389
q11	417	199	198	198
q12	741	271	280	271
q13	18020	3294	3294	3294
q14	281	247	253	247
q15	533	498	502	498
q16	469	411	411	411
q17	944	515	499	499
q18	6764	6121	5873	5873
q19	1567	1358	1442	1358
q20	586	332	327	327
q21	6833	3172	3153	3153
q22	815	295	296	295
Total cold run time: 106855 ms
Total hot run time: 37090 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4364	4326	4336	4326
q2	338	238	231	231
q3	2963	2831	2818	2818
q4	1813	1623	1664	1623
q5	5155	5254	5227	5227
q6	190	113	115	113
q7	2158	1783	1782	1782
q8	3173	3209	3198	3198
q9	8303	8318	8317	8317
q10	5777	3541	3589	3541
q11	542	466	462	462
q12	740	607	591	591
q13	7982	3105	3071	3071
q14	273	250	257	250
q15	538	492	489	489
q16	538	479	466	466
q17	1863	1720	1715	1715
q18	7849	7875	7618	7618
q19	7499	1537	1544	1537
q20	2144	1902	1912	1902
q21	4734	4622	4534	4534
q22	565	497	455	455
Total cold run time: 69501 ms
Total hot run time: 54266 ms

@doris-robot
Copy link

doris-robot commented Feb 7, 2024

TeamCity be ut coverage result:
Function Coverage: 36.17% (8638/23880)
Line Coverage: 28.20% (70625/250461)
Region Coverage: 27.18% (36435/134031)
Branch Coverage: 23.97% (18662/77844)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d28afd1b3c8e80f75a0120d50812084bb90a5d84_d28afd1b3c8e80f75a0120d50812084bb90a5d84/report/index.html

@doris-robot
Copy link

doris-robot commented Feb 7, 2024

TPC-DS: Total hot run time: 174259 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d28afd1b3c8e80f75a0120d50812084bb90a5d84, data reload: false

query1	927	343	342	342
query2	6528	1857	1845	1845
query3	6695	210	203	203
query4	31884	21982	22046	21982
query5	4243	423	421	421
query6	253	170	167	167
query7	4626	277	279	277
query8	254	172	181	172
query9	9024	2345	2314	2314
query10	404	226	219	219
query11	18970	15556	15285	15285
query12	129	77	79	77
query13	1622	429	414	414
query14	9241	7206	7234	7206
query15	247	175	185	175
query16	8145	254	241	241
query17	2034	547	491	491
query18	2103	260	258	258
query19	364	142	137	137
query20	82	76	76	76
query21	194	126	119	119
query22	4851	4590	4391	4391
query23	31898	29984	30096	29984
query24	11025	2722	2768	2722
query25	574	337	351	337
query26	1393	145	147	145
query27	2908	303	305	303
query28	7386	1863	1848	1848
query29	863	618	615	615
query30	279	136	138	136
query31	894	705	711	705
query32	95	54	49	49
query33	735	216	217	216
query34	1056	460	455	455
query35	859	782	771	771
query36	1021	918	933	918
query37	116	55	55	55
query38	3229	3113	3113	3113
query39	1285	1270	1229	1229
query40	185	92	91	91
query41	38	36	35	35
query42	99	88	96	88
query43	503	490	491	490
query44	1124	692	699	692
query45	194	184	181	181
query46	1052	658	668	658
query47	1589	1550	1444	1444
query48	410	377	350	350
query49	1098	283	285	283
query50	759	370	387	370
query51	5283	5145	5111	5111
query52	103	81	87	81
query53	349	268	276	268
query54	274	214	218	214
query55	86	81	72	72
query56	218	203	195	195
query57	995	917	897	897
query58	198	177	175	175
query59	2431	2314	2315	2314
query60	237	222	215	215
query61	82	87	81	81
query62	672	348	367	348
query63	298	269	268	268
query64	5539	3591	3658	3591
query65	3260	3246	3222	3222
query66	1001	309	307	307
query67	14598	14084	14072	14072
query68	4358	541	531	531
query69	462	312	319	312
query70	1256	1215	1186	1186
query71	328	255	242	242
query72	6046	2825	2668	2668
query73	710	320	323	320
query74	6599	6144	6205	6144
query75	3036	2370	2307	2307
query76	2906	963	953	953
query77	349	231	223	223
query78	9149	8638	8498	8498
query79	2964	493	504	493
query80	2022	342	342	342
query81	547	196	197	196
query82	837	86	76	76
query83	240	122	125	122
query84	285	84	76	76
query85	2202	341	322	322
query86	482	314	304	304
query87	3360	3213	3187	3187
query88	4276	2303	2320	2303
query89	444	374	353	353
query90	1934	164	161	161
query91	144	121	118	118
query92	56	43	45	43
query93	4862	466	488	466
query94	1234	179	171	171
query95	469	354	7883	354
query96	593	274	273	273
query97	4265	4128	4107	4107
query98	221	205	209	205
query99	1117	677	678	677
Total cold run time: 288864 ms
Total hot run time: 174259 ms

@doris-robot
Copy link

doris-robot commented Feb 7, 2024

ClickBench: Total hot run time: 30.93 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d28afd1b3c8e80f75a0120d50812084bb90a5d84, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.02	0.02
query3	0.22	0.06	0.06
query4	1.67	0.09	0.10
query5	0.54	0.51	0.52
query6	1.18	0.63	0.66
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.58	0.50	0.51
query10	0.55	0.54	0.54
query11	0.11	0.08	0.08
query12	0.10	0.08	0.09
query13	0.60	0.60	0.60
query14	0.78	0.81	0.81
query15	0.79	0.78	0.77
query16	0.41	0.40	0.38
query17	1.04	1.01	1.02
query18	0.21	0.27	0.25
query19	1.85	1.80	1.80
query20	0.02	0.01	0.01
query21	15.44	0.60	0.54
query22	2.55	2.07	2.27
query23	17.26	0.75	0.80
query24	3.04	0.63	0.85
query25	0.28	0.20	0.18
query26	0.48	0.13	0.14
query27	0.04	0.05	0.04
query28	11.47	0.84	0.84
query29	12.62	3.20	3.19
query30	0.65	0.52	0.53
query31	2.80	0.35	0.35
query32	3.34	0.48	0.48
query33	3.22	3.15	3.26
query34	15.72	4.28	4.30
query35	4.30	4.29	4.29
query36	1.08	1.05	1.06
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.13	0.13
query41	0.07	0.01	0.01
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.48 s
Total hot run time: 30.93 s

@doris-robot
Copy link

doris-robot commented Feb 7, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d28afd1b3c8e80f75a0120d50812084bb90a5d84 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.5 seconds inserted 10000000 Rows, about 689K ops/s

@github-actions
Copy link
Contributor

github-actions bot commented Feb 7, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Feb 7, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Feb 7, 2024

PR approved by anyone and no changes requested.

@xiaokang xiaokang mentioned this pull request Feb 7, 2024
Merged
airborne12
airborne12 approved these changes Feb 7, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang changed the title [feature](index) remove OR predicate support for bitmap index [fix](index) remove OR predicate support for bitmap index Feb 7, 2024
qidaye
qidaye approved these changes Feb 7, 2024
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qidaye qidaye merged commit 7ed7bba into apache:master Feb 7, 2024
yiguolei pushed a commit that referenced this pull request Feb 8, 2024
@xiaokang @Doris-Extras
disable OR predicate for bitmap index (#30951)
479ff98 
There is some problem for OR predicate push down using bitmap index, so disable it.
yiguolei pushed a commit that referenced this pull request Feb 16, 2024
@xiaokang @Doris-Extras
disable OR predicate for bitmap index (#30951)
abbd1c7 
There is some problem for OR predicate push down using bitmap index, so disable it.
mymeiyi pushed a commit to mymeiyi/doris that referenced this pull request Feb 19, 2024
@xiaokang @mymeiyi
disable OR predicate for bitmap index (apache#30951)
9f0812e 
There is some problem for OR predicate push down using bitmap index, so disable it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

@yiguolei yiguolei yiguolei approved these changes

@qidaye qidaye qidaye approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.5-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xiaokang @doris-robot @airborne12 @yiguolei @qidaye