Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix] row policy with a predicate containing non-existing column #32034 #32035

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

LuGuangming
Copy link
Contributor

Proposed changes

Issue Number: close #32034

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@LuGuangming
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37577 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4aa8d0808dab649aceb29ff06c075820236d13b2, data reload: false

------ Round 1 ----------------------------------
q1	17668	3966	3976	3966
q2	2015	144	135	135
q3	10583	985	921	921
q4	4663	911	934	911
q5	7602	2793	3009	2793
q6	179	127	128	127
q7	1267	802	799	799
q8	9487	2043	2050	2043
q9	7290	6419	6407	6407
q10	8200	2559	2563	2559
q11	424	226	220	220
q12	756	315	313	313
q13	17963	2928	2940	2928
q14	281	253	255	253
q15	475	446	434	434
q16	481	379	401	379
q17	937	886	865	865
q18	6572	5922	5792	5792
q19	1556	1501	1495	1495
q20	554	275	289	275
q21	7538	3648	3659	3648
q22	801	325	314	314
Total cold run time: 107292 ms
Total hot run time: 37577 ms

----- Round 2, with runtime_filter_mode=off -----
q1	3994	3977	3988	3977
q2	316	227	221	221
q3	2932	2879	2889	2879
q4	1823	1804	1773	1773
q5	5220	5259	5223	5223
q6	204	117	124	117
q7	2287	1833	1792	1792
q8	3191	3237	3252	3237
q9	8475	8482	8478	8478
q10	6123	3729	3712	3712
q11	533	439	434	434
q12	687	521	537	521
q13	11896	2828	2801	2801
q14	273	248	261	248
q15	466	429	432	429
q16	448	413	404	404
q17	1678	1646	1624	1624
q18	7753	7383	7173	7173
q19	1612	1596	1604	1596
q20	1933	1701	1723	1701
q21	4909	4799	4795	4795
q22	518	456	461	456
Total cold run time: 67271 ms
Total hot run time: 53591 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186452 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4aa8d0808dab649aceb29ff06c075820236d13b2, data reload: false

query1	918	374	335	335
query2	7363	2017	2073	2017
query3	7593	219	219	219
query4	26030	21056	21438	21056
query5	4391	523	556	523
query6	273	207	204	204
query7	4751	348	339	339
query8	256	192	197	192
query9	8636	2338	2328	2328
query10	452	269	296	269
query11	15109	14472	14397	14397
query12	133	89	90	89
query13	1643	431	421	421
query14	11718	10045	9576	9576
query15	276	214	194	194
query16	8143	277	298	277
query17	1723	631	566	566
query18	2120	298	278	278
query19	336	163	183	163
query20	99	88	88	88
query21	205	132	128	128
query22	4784	4481	4462	4462
query23	31278	31031	30491	30491
query24	10078	3152	3038	3038
query25	629	401	407	401
query26	871	164	162	162
query27	2302	365	374	365
query28	5728	1881	1877	1877
query29	953	625	608	608
query30	314	147	152	147
query31	954	725	743	725
query32	100	62	57	57
query33	767	335	267	267
query34	913	462	470	462
query35	853	636	627	627
query36	956	832	836	832
query37	105	77	76	76
query38	3326	3256	3206	3206
query39	1425	1388	1383	1383
query40	214	120	121	120
query41	63	57	55	55
query42	111	105	105	105
query43	438	392	412	392
query44	1081	733	720	720
query45	279	256	261	256
query46	1046	788	776	776
query47	1610	1548	1568	1548
query48	424	351	358	351
query49	1098	342	341	341
query50	784	382	384	382
query51	6838	6790	6773	6773
query52	103	95	90	90
query53	351	280	289	280
query54	310	264	249	249
query55	89	88	85	85
query56	257	244	262	244
query57	1087	980	985	980
query58	248	220	215	215
query59	2316	2270	2221	2221
query60	263	266	261	261
query61	119	117	119	117
query62	649	425	416	416
query63	316	284	285	284
query64	4967	3420	3055	3055
query65	3017	2996	2989	2989
query66	833	326	314	314
query67	15221	14450	14331	14331
query68	16619	560	582	560
query69	739	398	391	391
query70	1866	1112	1063	1063
query71	593	280	282	280
query72	10133	2652	2521	2521
query73	3880	342	348	342
query74	7195	6934	6757	6757
query75	12215	8124	7984	7984
query76	7233	1067	1169	1067
query77	1127	267	260	260
query78	10196	9597	9426	9426
query79	8374	513	507	507
query80	1451	386	401	386
query81	511	212	212	212
query82	339	218	212	212
query83	397	146	146	146
query84	291	77	77	77
query85	1257	358	329	329
query86	359	308	278	278
query87	3444	3176	3303	3176
query88	3196	2383	2370	2370
query89	526	361	360	360
query90	2540	184	179	179
query91	161	135	152	135
query92	63	50	59	50
query93	3700	544	527	527
query94	1689	196	195	195
query95	459	361	343	343
query96	591	265	266	265
query97	4430	4294	4324	4294
query98	220	214	215	214
query99	1051	751	764	751
Total cold run time: 319481 ms
Total hot run time: 186452 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4aa8d0808dab649aceb29ff06c075820236d13b2, data reload: false

query1	0.05	0.04	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.67	0.10	0.10
query5	0.50	0.48	0.50
query6	1.26	0.64	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.56	0.52	0.50
query10	0.56	0.57	0.56
query11	0.13	0.10	0.09
query12	0.12	0.10	0.10
query13	0.57	0.57	0.58
query14	0.72	0.75	0.75
query15	0.82	0.79	0.80
query16	0.38	0.38	0.36
query17	0.96	0.99	0.99
query18	0.27	0.26	0.27
query19	1.78	1.80	1.72
query20	0.02	0.01	0.02
query21	15.41	0.62	0.61
query22	3.28	5.06	2.81
query23	17.42	1.06	1.02
query24	2.13	0.77	0.39
query25	0.35	0.04	0.04
query26	0.17	0.13	0.13
query27	0.03	0.03	0.03
query28	11.96	0.82	0.83
query29	12.87	3.31	3.24
query30	0.59	0.60	0.59
query31	2.79	0.33	0.34
query32	3.40	0.44	0.43
query33	2.93	2.90	2.92
query34	15.50	4.32	4.34
query35	4.32	4.33	4.32
query36	1.08	1.02	1.03
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.12	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 105.37 s
Total hot run time: 31.35 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4aa8d0808dab649aceb29ff06c075820236d13b2 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       17.5 seconds inserted 10000000 Rows, about 571K ops/s

@xiaokang xiaokang added usercase Important user case type label dev/2.0.x labels Mar 11, 2024
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a testcase

@xiaokang
Copy link
Contributor

@LuGuangming can you add a testcase for this fix?

@xiaokang xiaokang removed the usercase Important user case type label label Jun 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] row policy with a predicate containing non-existing column
3 participants