Skip to content

[fix](autobucket) calc bucket num exclude today's partition #34304#34507

Merged
dataroaring merged 1 commit intoapache:branch-2.0from
yujun777:pick_34304_to_upstream_branch-2.0
May 8, 2024
Merged

[fix](autobucket) calc bucket num exclude today's partition #34304#34507
dataroaring merged 1 commit intoapache:branch-2.0from
yujun777:pick_34304_to_upstream_branch-2.0

Conversation

@yujun777
Copy link
Contributor

@yujun777 yujun777 commented May 8, 2024

cherry pick from #34304

@yujun777
Copy link
Contributor Author

yujun777 commented May 8, 2024

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yujun777 yujun777 marked this pull request as draft May 8, 2024 02:56
@doris-robot
Copy link

TPC-H: Total hot run time: 49609 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 77fb8eb42b615d9de9f0ae9add5a22c8bc044af6, data reload: false

------ Round 1 ----------------------------------
q1	17859	4353	4339	4339
q2	2029	151	139	139
q3	10416	1929	1913	1913
q4	10330	1248	1335	1248
q5	8316	3879	3906	3879
q6	231	121	122	121
q7	2009	1608	1608	1608
q8	9488	2703	2707	2703
q9	10388	10384	10214	10214
q10	8634	3523	3512	3512
q11	419	236	240	236
q12	466	305	308	305
q13	18338	3978	3979	3978
q14	352	314	316	314
q15	513	467	452	452
q16	670	573	553	553
q17	1123	977	998	977
q18	7174	6939	6884	6884
q19	1692	1583	1518	1518
q20	513	312	300	300
q21	4433	4062	4034	4034
q22	485	382	391	382
Total cold run time: 115878 ms
Total hot run time: 49609 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4284	4272	4273	4272
q2	315	222	220	220
q3	4148	4153	4136	4136
q4	2742	2736	2725	2725
q5	7213	7103	7116	7103
q6	232	118	117	117
q7	3230	2785	2805	2785
q8	4348	4551	4441	4441
q9	16919	16730	16839	16730
q10	4221	4262	4256	4256
q11	737	684	701	684
q12	1032	862	838	838
q13	7131	3707	3773	3707
q14	464	438	422	422
q15	488	452	449	449
q16	744	684	671	671
q17	3808	3871	3770	3770
q18	8927	8715	8837	8715
q19	1692	1691	1647	1647
q20	2342	2177	2119	2119
q21	8594	8449	8485	8449
q22	1020	935	927	927
Total cold run time: 84631 ms
Total hot run time: 79183 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203245 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 77fb8eb42b615d9de9f0ae9add5a22c8bc044af6, data reload: false

query1	924	382	383	382
query2	6538	2765	2456	2456
query3	6917	203	198	198
query4	21461	17954	18113	17954
query5	19985	6549	6528	6528
query6	289	217	246	217
query7	4161	307	312	307
query8	275	239	259	239
query9	3148	2725	2653	2653
query10	407	286	312	286
query11	11476	10896	10909	10896
query12	127	71	73	71
query13	6096	728	695	695
query14	19545	13719	13893	13719
query15	385	255	255	255
query16	7091	301	267	267
query17	1840	1447	896	896
query18	2344	419	422	419
query19	202	158	154	154
query20	83	75	75	75
query21	189	88	92	88
query22	5116	4984	5038	4984
query23	32718	31928	31997	31928
query24	6906	6573	6461	6461
query25	514	428	412	412
query26	511	167	160	160
query27	1803	296	299	296
query28	6137	2399	2351	2351
query29	2794	2826	2678	2678
query30	240	166	164	164
query31	901	727	726	726
query32	66	62	63	62
query33	415	248	253	248
query34	839	472	477	472
query35	1105	918	937	918
query36	1217	1258	1195	1195
query37	87	63	64	63
query38	3052	2900	2951	2900
query39	1398	1312	1319	1312
query40	213	104	97	97
query41	39	36	35	35
query42	89	82	79	79
query43	815	692	766	692
query44	1146	745	738	738
query45	247	225	228	225
query46	1210	962	966	962
query47	1820	1789	1705	1705
query48	1031	743	726	726
query49	618	363	375	363
query50	874	588	570	570
query51	4802	4685	4636	4636
query52	92	80	85	80
query53	439	326	316	316
query54	2634	2465	2456	2456
query55	103	79	83	79
query56	215	211	210	210
query57	1168	1133	1061	1061
query58	220	199	202	199
query59	4478	3829	3732	3732
query60	209	210	205	205
query61	88	87	86	86
query62	843	559	494	494
query63	486	332	348	332
query64	2220	1482	1444	1444
query65	3658	3564	3538	3538
query66	758	374	385	374
query67	15712	15531	14859	14859
query68	9454	670	658	658
query69	576	355	367	355
query70	1661	1461	1456	1456
query71	429	291	307	291
query72	6468	3474	3473	3473
query73	741	328	316	316
query74	6378	6263	6096	6096
query75	5409	3697	3732	3697
query76	5889	1159	1213	1159
query77	997	262	254	254
query78	12634	11548	11601	11548
query79	6919	665	642	642
query80	977	414	406	406
query81	496	227	229	227
query82	1500	96	104	96
query83	176	145	127	127
query84	261	70	68	68
query85	852	301	296	296
query86	331	300	297	297
query87	3276	2988	3051	2988
query88	4921	2319	2313	2313
query89	375	273	287	273
query90	1944	214	215	214
query91	164	135	137	135
query92	55	52	55	52
query93	4836	555	579	555
query94	702	205	209	205
query95	1130	1062	1068	1062
query96	647	344	325	325
query97	6416	6289	6392	6289
query98	192	180	166	166
query99	3009	828	830	828
Total cold run time: 315344 ms
Total hot run time: 203245 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.46 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 77fb8eb42b615d9de9f0ae9add5a22c8bc044af6, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.05
query4	1.79	0.08	0.08
query5	0.53	0.53	0.52
query6	1.33	0.62	0.63
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.54	0.48	0.48
query10	0.53	0.53	0.54
query11	0.12	0.08	0.09
query12	0.11	0.09	0.10
query13	0.61	0.62	0.61
query14	0.79	0.78	0.80
query15	0.78	0.77	0.76
query16	0.37	0.37	0.38
query17	1.03	1.00	1.02
query18	0.23	0.25	0.24
query19	1.90	1.81	1.83
query20	0.01	0.01	0.01
query21	15.49	0.53	0.56
query22	2.01	2.18	1.43
query23	17.41	1.00	0.92
query24	7.57	1.31	1.00
query25	0.35	0.15	0.05
query26	0.85	0.17	0.17
query27	0.04	0.03	0.04
query28	5.35	0.79	0.73
query29	12.68	2.18	2.31
query30	0.59	0.54	0.51
query31	2.83	0.39	0.38
query32	3.39	0.49	0.50
query33	3.01	3.11	3.10
query34	15.28	4.77	4.80
query35	4.86	4.85	4.82
query36	1.06	1.03	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 104.23 s
Total hot run time: 30.46 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 77fb8eb42b615d9de9f0ae9add5a22c8bc044af6 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

@yujun777 yujun777 marked this pull request as ready for review May 8, 2024 06:11
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit ca7fb51 into apache:branch-2.0 May 8, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants