Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](nereids)update session var for broadcast penalty #40199

Merged
merged 2 commits into from
Sep 2, 2024

Conversation

englefly
Copy link
Contributor

Proposed changes

pr25421 pick pr22092 from master to branch-2.0, but forget to update session variable BROADCAST_RIGHT_TABLE_SCALE_FACTOR
Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

run buildall

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 49089 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 849f2ae8b4cefc0d1ae629145e643d932f945258, data reload: false

------ Round 1 ----------------------------------
q1	18161	4356	4341	4341
q2	2071	155	146	146
q3	10260	1897	1951	1897
q4	10282	1213	1313	1213
q5	8681	3833	3845	3833
q6	224	124	119	119
q7	2050	1623	1618	1618
q8	9308	2739	2702	2702
q9	10221	9855	9861	9855
q10	8671	3588	3524	3524
q11	420	244	245	244
q12	467	290	292	290
q13	18383	3963	4026	3963
q14	363	326	325	325
q15	511	467	446	446
q16	569	445	458	445
q17	1135	970	960	960
q18	7224	6864	6899	6864
q19	1663	1571	1502	1502
q20	539	303	309	303
q21	4411	4109	4099	4099
q22	489	400	409	400
Total cold run time: 116103 ms
Total hot run time: 49089 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4337	4294	4350	4294
q2	327	227	224	224
q3	4136	4130	4164	4130
q4	2756	2743	2747	2743
q5	7138	7052	7109	7052
q6	236	115	117	115
q7	3274	2892	2819	2819
q8	4387	4501	4486	4486
q9	13690	13555	13518	13518
q10	4241	4275	4280	4275
q11	781	688	664	664
q12	1044	851	859	851
q13	6685	3733	3724	3724
q14	458	422	427	422
q15	501	461	458	458
q16	625	625	578	578
q17	3883	3799	3808	3799
q18	8850	8725	8785	8725
q19	1728	1655	1657	1655
q20	2379	2116	2091	2091
q21	8610	8631	8547	8547
q22	1055	937	989	937
Total cold run time: 81121 ms
Total hot run time: 76107 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 212274 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 849f2ae8b4cefc0d1ae629145e643d932f945258, data reload: false

query1	922	386	415	386
query2	6617	2115	2256	2115
query3	6914	201	201	201
query4	23516	22458	22014	22014
query5	21024	6587	6620	6587
query6	1067	226	248	226
query7	6852	325	326	325
query8	342	263	271	263
query9	5433	2684	2653	2653
query10	469	295	308	295
query11	15735	15233	14761	14761
query12	120	75	77	75
query13	1007	442	444	442
query14	19048	13555	13192	13192
query15	370	219	232	219
query16	6434	285	267	267
query17	1521	939	897	897
query18	898	312	319	312
query19	208	152	154	152
query20	82	77	81	77
query21	184	95	92	92
query22	5165	4966	5002	4966
query23	33992	33513	33413	33413
query24	6502	6362	6318	6318
query25	513	414	431	414
query26	1004	162	162	162
query27	2240	296	306	296
query28	6095	2307	2254	2254
query29	2874	2612	2801	2612
query30	246	169	167	167
query31	950	745	749	745
query32	71	68	65	65
query33	440	249	253	249
query34	890	473	482	473
query35	1132	928	930	928
query36	1300	1050	1161	1050
query37	89	60	60	60
query38	3117	2949	2931	2931
query39	1392	1322	1304	1304
query40	212	96	95	95
query41	40	38	37	37
query42	89	83	83	83
query43	679	611	617	611
query44	1157	729	733	729
query45	243	227	235	227
query46	1229	971	947	947
query47	2071	1728	1804	1728
query48	511	420	410	410
query49	617	361	369	361
query50	861	607	580	580
query51	4757	4685	4713	4685
query52	94	80	84	80
query53	228	186	182	182
query54	2684	2479	2516	2479
query55	89	83	87	83
query56	227	217	203	203
query57	1230	1171	1105	1105
query58	215	213	214	213
query59	3414	3479	3311	3311
query60	213	211	199	199
query61	96	93	96	93
query62	817	531	520	520
query63	203	176	179	176
query64	3296	1559	1490	1490
query65	3620	3546	3576	3546
query66	777	400	410	400
query67	15810	16842	15178	15178
query68	9537	653	650	650
query69	494	263	253	253
query70	1654	1364	1426	1364
query71	424	293	314	293
query72	6834	4765	4847	4765
query73	744	326	324	324
query74	6272	5859	5902	5859
query75	5404	3670	3727	3670
query76	5368	1145	1184	1145
query77	940	249	258	249
query78	12563	11515	11473	11473
query79	7890	631	635	631
query80	1578	379	383	379
query81	485	239	236	236
query82	1680	96	96	96
query83	174	131	132	131
query84	251	70	70	70
query85	905	318	317	317
query86	330	296	280	280
query87	3229	3016	3026	3016
query88	4709	2296	2304	2296
query89	445	291	294	291
query90	2058	203	213	203
query91	162	129	140	129
query92	61	52	56	52
query93	6296	590	568	568
query94	744	207	207	207
query95	1991	1950	1934	1934
query96	658	338	328	328
query97	6488	6371	6439	6371
query98	219	206	213	206
query99	3046	779	893	779
Total cold run time: 325887 ms
Total hot run time: 212274 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.51 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 849f2ae8b4cefc0d1ae629145e643d932f945258, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.02
query3	0.25	0.04	0.04
query4	1.80	0.06	0.05
query5	0.53	0.52	0.52
query6	1.28	0.60	0.65
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.50	0.48
query10	0.54	0.52	0.53
query11	0.11	0.08	0.08
query12	0.12	0.09	0.09
query13	0.64	0.62	0.61
query14	0.78	0.79	0.78
query15	0.76	0.76	0.75
query16	0.39	0.37	0.39
query17	1.02	1.01	1.00
query18	0.20	0.28	0.21
query19	1.93	1.87	1.81
query20	0.02	0.01	0.01
query21	15.45	0.56	0.53
query22	2.06	2.06	1.55
query23	17.32	0.96	0.92
query24	6.09	2.81	0.92
query25	0.38	0.11	0.05
query26	0.86	0.15	0.15
query27	0.04	0.04	0.03
query28	5.32	0.74	0.75
query29	12.61	2.27	2.35
query30	0.58	0.52	0.53
query31	2.82	0.38	0.37
query32	3.35	0.50	0.49
query33	3.04	3.08	3.06
query34	15.25	4.78	4.78
query35	4.87	4.85	4.87
query36	1.04	1.02	1.02
query37	0.06	0.05	0.05
query38	0.04	0.03	0.02
query39	0.02	0.02	0.02
query40	0.15	0.14	0.15
query41	0.06	0.02	0.01
query42	0.03	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 102.49 s
Total hot run time: 30.51 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 849f2ae8b4cefc0d1ae629145e643d932f945258 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@englefly
Copy link
Contributor Author

englefly commented Sep 2, 2024

run p0

@englefly
Copy link
Contributor Author

englefly commented Sep 2, 2024

run feut

@englefly
Copy link
Contributor Author

englefly commented Sep 2, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49194 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 530de7820a5d67419afe78a8de80e385bae8560a, data reload: false

------ Round 1 ----------------------------------
q1	17631	4473	4383	4383
q2	2094	151	148	148
q3	10469	1962	1887	1887
q4	10366	1304	1329	1304
q5	8541	3915	3903	3903
q6	241	125	122	122
q7	2052	1637	1609	1609
q8	9585	2770	2732	2732
q9	12566	9840	9676	9676
q10	8711	3552	3568	3552
q11	408	251	241	241
q12	463	296	298	296
q13	18315	3979	4057	3979
q14	348	322	332	322
q15	506	456	458	456
q16	523	461	462	461
q17	1135	978	971	971
q18	7190	6860	6990	6860
q19	1679	1551	1540	1540
q20	516	319	288	288
q21	4465	4188	4072	4072
q22	483	392	397	392
Total cold run time: 118287 ms
Total hot run time: 49194 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4381	4324	4340	4324
q2	324	228	220	220
q3	4138	4123	4170	4123
q4	2748	2748	2753	2748
q5	7212	7189	7080	7080
q6	240	118	117	117
q7	3206	2818	2862	2818
q8	4332	4461	4490	4461
q9	13788	13631	13512	13512
q10	4245	4244	4254	4244
q11	762	705	703	703
q12	1028	853	846	846
q13	6740	3737	3751	3737
q14	446	427	425	425
q15	497	457	457	457
q16	643	584	584	584
q17	3833	3903	3828	3828
q18	8871	8695	8689	8689
q19	1720	1687	1664	1664
q20	2422	2146	2096	2096
q21	8577	8535	8510	8510
q22	1038	961	986	961
Total cold run time: 81191 ms
Total hot run time: 76147 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 212256 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 530de7820a5d67419afe78a8de80e385bae8560a, data reload: false

query1	955	392	409	392
query2	6552	2031	2182	2031
query3	6918	209	204	204
query4	23318	21597	21696	21597
query5	19731	6521	6564	6521
query6	280	235	231	231
query7	4213	301	308	301
query8	246	259	269	259
query9	3065	2645	2588	2588
query10	436	297	300	297
query11	15970	15321	14870	14870
query12	119	81	75	75
query13	1032	437	433	433
query14	17067	12989	13528	12989
query15	383	218	234	218
query16	6511	284	270	270
query17	1737	942	882	882
query18	893	311	320	311
query19	208	154	151	151
query20	79	78	80	78
query21	193	100	95	95
query22	5387	5276	5013	5013
query23	34276	33507	33581	33507
query24	7953	6364	6346	6346
query25	522	431	427	427
query26	1274	166	166	166
query27	2508	300	297	297
query28	6048	2251	2231	2231
query29	2892	2859	2833	2833
query30	250	167	169	167
query31	935	718	743	718
query32	71	66	62	62
query33	461	267	267	267
query34	873	476	477	476
query35	1136	938	931	931
query36	1184	1352	1157	1157
query37	174	60	60	60
query38	3049	2895	2883	2883
query39	1369	1322	1343	1322
query40	305	96	89	89
query41	40	39	40	39
query42	82	81	84	81
query43	605	590	679	590
query44	1146	728	730	728
query45	244	236	228	228
query46	1232	962	968	962
query47	1859	1691	1710	1691
query48	516	422	410	410
query49	647	377	378	377
query50	867	603	639	603
query51	4776	4641	4662	4641
query52	90	75	80	75
query53	233	187	182	182
query54	2691	2499	2456	2456
query55	100	91	77	77
query56	227	231	208	208
query57	1220	1232	1018	1018
query58	212	206	219	206
query59	3421	3226	3380	3226
query60	213	209	214	209
query61	97	95	99	95
query62	846	472	470	470
query63	196	174	176	174
query64	3573	1616	1472	1472
query65	3609	3582	3545	3545
query66	772	414	431	414
query67	15331	15218	15695	15218
query68	9509	682	661	661
query69	486	265	254	254
query70	1617	1527	1586	1527
query71	399	292	309	292
query72	6823	4789	4850	4789
query73	750	323	330	323
query74	6275	5893	5904	5893
query75	5117	3712	3629	3629
query76	5147	1155	1056	1056
query77	913	256	258	256
query78	12740	12074	12293	12074
query79	8835	622	638	622
query80	1173	382	385	382
query81	487	243	234	234
query82	1438	95	96	95
query83	172	131	133	131
query84	261	71	68	68
query85	998	324	320	320
query86	338	297	285	285
query87	3284	3072	3009	3009
query88	4621	2316	2287	2287
query89	461	291	307	291
query90	1907	221	215	215
query91	159	129	127	127
query92	61	53	52	52
query93	5296	595	550	550
query94	732	209	202	202
query95	2090	1756	1899	1756
query96	649	332	331	331
query97	6445	6355	6379	6355
query98	231	210	203	203
query99	2774	847	891	847
Total cold run time: 317606 ms
Total hot run time: 212256 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.47 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 530de7820a5d67419afe78a8de80e385bae8560a, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.24	0.04	0.05
query4	1.80	0.07	0.06
query5	0.53	0.53	0.53
query6	1.25	0.62	0.61
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.52	0.48	0.48
query10	0.53	0.51	0.53
query11	0.13	0.08	0.08
query12	0.12	0.09	0.09
query13	0.61	0.62	0.61
query14	0.80	0.80	0.80
query15	0.78	0.77	0.75
query16	0.37	0.40	0.36
query17	1.04	1.03	1.02
query18	0.24	0.24	0.23
query19	1.92	1.90	1.90
query20	0.02	0.01	0.01
query21	15.49	0.55	0.55
query22	2.04	2.60	1.89
query23	17.52	0.94	0.90
query24	3.63	2.39	1.40
query25	0.35	0.10	0.05
query26	0.63	0.17	0.18
query27	0.04	0.04	0.04
query28	7.80	0.73	0.74
query29	12.63	2.34	2.31
query30	0.61	0.50	0.53
query31	2.81	0.40	0.36
query32	3.40	0.49	0.50
query33	3.10	3.05	3.09
query34	15.25	4.80	4.80
query35	4.84	4.85	4.83
query36	1.06	1.01	1.01
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.03	0.01	0.01
Total cold run time: 102.65 s
Total hot run time: 31.47 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 530de7820a5d67419afe78a8de80e385bae8560a with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@englefly englefly merged commit 3ee0e2b into apache:branch-2.0 Sep 2, 2024
21 of 23 checks passed
@englefly englefly deleted the bc-var branch September 2, 2024 06:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants