Skip to content

Conversation

@airborne12
Copy link
Member

Proposed changes

The optimization to avoid reading data for primary keys (PK) in segment_iterator can lead to incorrect results when there are function expressions involving the same PK column in the predicate.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@airborne12
Copy link
Member Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.48% (9009/24695)
Line Coverage: 28.03% (73886/263632)
Region Coverage: 27.51% (38367/139489)
Branch Coverage: 24.21% (19556/80792)
Coverage Report: http://coverage.selectdb-in.cc/coverage/39e67a83de4b3ae38f330c7ed66d3402da30a3e2_39e67a83de4b3ae38f330c7ed66d3402da30a3e2/report/index.html

Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 20, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@airborne12 airborne12 force-pushed the fix-pk-no-need-read-data branch from 39e67a8 to e1d3ec4 Compare June 20, 2024 14:15
@airborne12
Copy link
Member Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40088 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e1d3ec4089b66913117191c89ec774373896bcd9, data reload: false

------ Round 1 ----------------------------------
q1	5970	4302	4264	4264
q2	1635	198	197	197
q3	1410	1091	1208	1091
q4	1382	794	770	770
q5	2658	2688	2622	2622
q6	223	138	138	138
q7	975	614	600	600
q8	2078	2366	2130	2130
q9	8940	6539	6460	6460
q10	8952	3740	3735	3735
q11	439	246	235	235
q12	399	243	222	222
q13	17753	2968	2961	2961
q14	287	223	221	221
q15	505	476	475	475
q16	486	395	380	380
q17	977	686	696	686
q18	8069	7670	7533	7533
q19	7642	1441	1451	1441
q20	653	321	333	321
q21	4944	3268	3957	3268
q22	401	338	343	338
Total cold run time: 76778 ms
Total hot run time: 40088 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4316	4249	4244	4244
q2	372	264	271	264
q3	2966	2726	2735	2726
q4	1889	1581	1598	1581
q5	5292	5326	5301	5301
q6	214	127	131	127
q7	2105	1777	1741	1741
q8	3210	3357	3339	3339
q9	8411	8352	8424	8352
q10	3882	3677	3719	3677
q11	572	496	479	479
q12	772	611	617	611
q13	16395	2969	2982	2969
q14	296	274	271	271
q15	517	475	480	475
q16	453	405	424	405
q17	1787	1482	1466	1466
q18	7651	7614	7328	7328
q19	1700	1630	1680	1630
q20	1992	1784	1774	1774
q21	4835	4796	4856	4796
q22	607	541	554	541
Total cold run time: 70234 ms
Total hot run time: 54097 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.48% (9009/24695)
Line Coverage: 28.03% (73894/263632)
Region Coverage: 27.52% (38381/139489)
Branch Coverage: 24.21% (19562/80792)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e1d3ec4089b66913117191c89ec774373896bcd9_e1d3ec4089b66913117191c89ec774373896bcd9/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 172194 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e1d3ec4089b66913117191c89ec774373896bcd9, data reload: false

query1	915	392	391	391
query2	6474	2384	2286	2286
query3	6648	207	209	207
query4	19407	17369	17153	17153
query5	4159	503	472	472
query6	249	164	167	164
query7	4593	303	291	291
query8	332	288	282	282
query9	8641	2443	2434	2434
query10	610	317	287	287
query11	10389	10132	10012	10012
query12	146	93	88	88
query13	1663	393	356	356
query14	9675	7659	6863	6863
query15	232	189	189	189
query16	7840	265	265	265
query17	1889	538	555	538
query18	1970	283	282	282
query19	200	160	156	156
query20	94	82	86	82
query21	211	128	126	126
query22	4381	4061	3904	3904
query23	33645	32876	32908	32876
query24	10998	2878	2875	2875
query25	614	364	357	357
query26	1193	153	155	153
query27	2533	325	317	317
query28	7112	2073	2091	2073
query29	877	620	616	616
query30	273	149	151	149
query31	941	746	793	746
query32	100	54	56	54
query33	776	295	290	290
query34	938	474	468	468
query35	753	656	604	604
query36	1125	968	937	937
query37	159	73	75	73
query38	2888	2741	2698	2698
query39	858	794	767	767
query40	204	133	130	130
query41	51	52	53	52
query42	126	105	98	98
query43	599	540	535	535
query44	1267	730	742	730
query45	198	166	168	166
query46	1069	714	731	714
query47	1836	1789	1789	1789
query48	371	298	297	297
query49	1112	432	443	432
query50	763	394	384	384
query51	6857	6847	6751	6751
query52	115	95	97	95
query53	357	299	296	296
query54	968	440	452	440
query55	76	77	74	74
query56	290	270	285	270
query57	1133	1073	1075	1073
query58	250	238	245	238
query59	3269	2964	2924	2924
query60	308	274	284	274
query61	95	94	103	94
query62	633	451	451	451
query63	321	291	304	291
query64	9032	2316	1744	1744
query65	3165	3124	3108	3108
query66	848	331	342	331
query67	15344	14837	14922	14837
query68	4540	547	524	524
query69	465	305	324	305
query70	1197	1133	1162	1133
query71	401	280	276	276
query72	7158	5368	5442	5368
query73	740	322	335	322
query74	5946	5573	5460	5460
query75	3464	2641	2648	2641
query76	2574	924	888	888
query77	514	320	313	313
query78	10427	9832	9810	9810
query79	2645	524	511	511
query80	2089	501	471	471
query81	594	218	215	215
query82	1033	111	112	111
query83	300	181	176	176
query84	265	93	90	90
query85	1230	292	277	277
query86	472	315	335	315
query87	3271	3110	3075	3075
query88	3757	2362	2343	2343
query89	481	386	388	386
query90	1743	191	193	191
query91	135	98	103	98
query92	71	51	51	51
query93	1738	515	499	499
query94	1199	195	193	193
query95	410	325	330	325
query96	589	268	262	262
query97	3189	3077	3029	3029
query98	217	205	198	198
query99	1163	872	839	839
Total cold run time: 271175 ms
Total hot run time: 172194 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.12 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e1d3ec4089b66913117191c89ec774373896bcd9, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.08
query5	0.50	0.47	0.48
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.52	0.49	0.50
query10	0.54	0.55	0.54
query11	0.16	0.12	0.11
query12	0.15	0.13	0.12
query13	0.60	0.59	0.60
query14	0.76	0.78	0.83
query15	0.85	0.80	0.80
query16	0.37	0.35	0.37
query17	0.99	1.02	1.03
query18	0.21	0.26	0.24
query19	1.76	1.81	1.72
query20	0.01	0.02	0.01
query21	15.41	0.68	0.67
query22	4.44	6.43	2.25
query23	18.31	1.43	1.36
query24	2.16	0.23	0.23
query25	0.17	0.10	0.08
query26	0.27	0.18	0.18
query27	0.09	0.08	0.09
query28	13.23	1.02	0.99
query29	12.59	3.31	3.28
query30	0.26	0.06	0.06
query31	2.86	0.38	0.38
query32	3.31	0.47	0.46
query33	2.93	2.92	2.87
query34	17.11	4.43	4.44
query35	4.51	4.56	4.51
query36	0.65	0.46	0.49
query37	0.19	0.16	0.16
query38	0.15	0.16	0.15
query39	0.04	0.04	0.04
query40	0.17	0.14	0.14
query41	0.11	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.7 s
Total hot run time: 31.12 s

Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qidaye qidaye merged commit 28f7ed3 into apache:master Jun 21, 2024
@airborne12 airborne12 deleted the fix-pk-no-need-read-data branch June 21, 2024 07:16
dataroaring pushed a commit that referenced this pull request Jun 21, 2024
The optimization to avoid reading data for primary keys (PK) in
segment_iterator can lead to incorrect results when there are function
expressions involving the same PK column in the predicate.
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants