[improvement](binlog) Change BinlogConfig default TTL_SECONDS to 86400 (1day) (#30771) by JackDrogon · Pull Request #30798 · apache/doris

JackDrogon · 2024-02-04T07:10:43Z

cherry-pick #30771

…0 (1day) (apache#30771) * Change BinlogConfig default TTL_SECONDS to 86400 (1day) Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com> * Fix binlog.ttl_seconds in regression test Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com> --------- Signed-off-by: Jack Drogon <jack.xsuperman@gmail.com>

doris-robot · 2024-02-04T07:10:48Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

dataroaring · 2024-02-04T07:56:32Z

run buildall

doris-robot · 2024-02-04T08:15:04Z

TPC-H: Total hot run time: 50311 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit acb9313e8371c6bd809a757ece6241e3d830dcd7, data reload: false

------ Round 1 ----------------------------------
q1	17968	4366	4348	4348
q2	2021	149	150	149
q3	10348	1888	1942	1888
q4	10108	1219	1298	1219
q5	8407	3951	4025	3951
q6	225	126	123	123
q7	2035	1633	1571	1571
q8	9298	2732	2728	2728
q9	11258	10618	10593	10593
q10	8636	3505	3538	3505
q11	413	235	244	235
q12	461	286	289	286
q13	18401	4035	4107	4035
q14	351	339	331	331
q15	502	468	466	466
q16	710	611	614	611
q17	1153	982	986	982
q18	7302	6977	7031	6977
q19	1666	1593	1525	1525
q20	533	320	307	307
q21	4464	4081	4113	4081
q22	504	412	400	400
Total cold run time: 116764 ms
Total hot run time: 50311 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4327	4320	4301	4301
q2	322	225	226	225
q3	4143	4148	4130	4130
q4	2739	2723	2733	2723
q5	7317	7205	7222	7205
q6	237	119	113	113
q7	3256	2881	2845	2845
q8	4484	4490	4551	4490
q9	17020	16999	17117	16999
q10	4178	4274	4296	4274
q11	738	697	681	681
q12	1017	837	863	837
q13	7311	3769	3705	3705
q14	462	429	416	416
q15	503	461	449	449
q16	748	711	700	700
q17	3847	3900	3830	3830
q18	8870	8935	8898	8898
q19	1701	1687	1661	1661
q20	2412	2115	2092	2092
q21	8604	8511	8418	8418
q22	1041	919	942	919
Total cold run time: 85277 ms
Total hot run time: 79911 ms

doris-robot · 2024-02-04T08:29:30Z

TPC-DS: Total hot run time: 236189 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit acb9313e8371c6bd809a757ece6241e3d830dcd7, data reload: false

query1	916	394	382	382
query2	6526	2262	2244	2244
query3	6937	210	207	207
query4	20048	17805	17931	17805
query5	19731	6242	6233	6233
query6	284	215	219	215
query7	4165	310	303	303
query8	254	233	249	233
query9	3127	2723	2663	2663
query10	418	298	312	298
query11	11326	10656	10590	10590
query12	123	73	72	72
query13	5631	644	642	642
query14	17426	13088	13364	13088
query15	356	238	241	238
query16	6443	258	259	258
query17	3210	876	859	859
query18	2261	405	403	403
query19	201	147	147	147
query20	77	76	80	76
query21	182	99	89	89
query22	5324	5189	5059	5059
query23	32454	32008	31758	31758
query24	7068	6468	6567	6468
query25	520	425	415	415
query26	602	162	162	162
query27	1991	305	294	294
query28	6116	2256	2200	2200
query29	3020	2713	2724	2713
query30	248	165	162	162
query31	912	711	728	711
query32	63	62	60	60
query33	394	244	256	244
query34	852	461	484	461
query35	1110	915	819	819
query36	1570	1673	1408	1408
query37	90	62	57	57
query38	3105	2919	2930	2919
query39	1368	1325	1307	1307
query40	202	92	97	92
query41	34	31	32	31
query42	94	94	81	81
query43	724	603	605	603
query44	1126	724	718	718
query45	245	230	228	228
query46	1250	960	956	956
query47	1998	1815	1814	1814
query48	974	672	664	664
query49	617	376	358	358
query50	857	623	609	609
query51	5531	5516	5502	5502
query52	77	82	84	82
query53	446	317	312	312
query54	2422	2231	2230	2230
query55	83	72	83	72
query56	199	205	196	196
query57	1140	1076	1118	1076
query58	214	181	191	181
query59	3367	3319	3158	3158
query60	208	185	196	185
query61	85	89	84	84
query62	737	435	437	435
query63	467	342	331	331
query64	2015	1288	1311	1288
query65	3686	3633	3628	3628
query66	823	376	362	362
query67	15679	15255	14644	14644
query68	11557	667	677	667
query69	563	355	343	343
query70	2336	1835	1875	1835
query71	386	305	323	305
query72	4549	3333	3346	3333
query73	1536	323	323	323
query74	6305	5949	5869	5869
query75	4927	3510	3529	3510
query76	7028	1167	1188	1167
query77	1176	259	258	258
query78	31416	51645	46760	46760
query79	14983	648	652	648
query80	4380	382	377	377
query81	560	241	244	241
query82	605	91	98	91
query83	404	138	127	127
query84	257	67	68	67
query85	2350	282	280	280
query86	462	356	369	356
query87	3273	2974	3061	2974
query88	6544	2383	2367	2367
query89	378	290	291	290
query90	2506	205	208	205
query91	161	142	121	121
query92	59	52	48	48
query93	4214	544	602	544
query94	1684	197	201	197
query95	1118	1074	1068	1068
query96	634	336	329	329
query97	6413	6402	6264	6264
query98	201	173	179	173
query99	3822	871	870	870
Total cold run time: 348536 ms
Total hot run time: 236189 ms

doris-robot · 2024-02-04T08:34:54Z

ClickBench: Total hot run time: 30.86 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit acb9313e8371c6bd809a757ece6241e3d830dcd7, data reload: false

query1	0.03	0.02	0.02
query2	0.05	0.03	0.02
query3	0.25	0.04	0.04
query4	1.85	0.07	0.06
query5	0.52	0.51	0.52
query6	1.30	0.60	0.61
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.49	0.49
query10	0.54	0.54	0.54
query11	0.11	0.09	0.09
query12	0.11	0.09	0.09
query13	0.62	0.61	0.61
query14	0.79	0.79	0.79
query15	0.77	0.77	0.78
query16	0.39	0.39	0.37
query17	1.02	1.01	1.02
query18	0.24	0.25	0.25
query19	1.93	1.84	1.78
query20	0.01	0.01	0.01
query21	15.47	0.54	0.54
query22	2.00	2.59	1.53
query23	17.18	1.04	0.98
query24	6.10	1.22	1.30
query25	1.61	0.12	0.12
query26	0.31	0.14	0.14
query27	0.11	0.11	0.10
query28	6.44	0.75	0.72
query29	12.78	2.31	2.15
query30	0.62	0.53	0.46
query31	2.83	0.38	0.39
query32	3.41	0.51	0.50
query33	3.09	3.05	3.12
query34	15.26	4.79	4.79
query35	4.86	4.85	4.86
query36	1.07	1.02	1.01
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.13
query41	0.06	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 104.63 s
Total hot run time: 30.86 s

doris-robot · 2024-02-04T08:38:06Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit acb9313e8371c6bd809a757ece6241e3d830dcd7 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

dataroaring

LGTM

github-actions · 2024-02-05T02:29:01Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-02-05T02:29:03Z

PR approved by anyone and no changes requested.

dataroaring · 2024-02-05T06:45:34Z

run buildall

doris-robot · 2024-02-05T07:16:03Z

TPC-H: Total hot run time: 49717 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e3cb12e6c6e882df8abebcd5ddd678317aef40be, data reload: false

------ Round 1 ----------------------------------
q1	17612	4362	4320	4320
q2	2071	157	141	141
q3	10441	1915	1911	1911
q4	10318	1251	1320	1251
q5	8415	3950	3989	3950
q6	223	122	121	121
q7	2047	1605	1619	1605
q8	9283	2688	2696	2688
q9	10732	10304	10373	10304
q10	8618	3530	3512	3512
q11	417	244	238	238
q12	463	306	303	303
q13	18356	3946	4006	3946
q14	339	324	324	324
q15	514	460	459	459
q16	684	591	590	590
q17	1116	946	955	946
q18	7211	6884	6857	6857
q19	1679	1560	1506	1506
q20	505	308	317	308
q21	4432	4038	4115	4038
q22	474	399	409	399
Total cold run time: 115950 ms
Total hot run time: 49717 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4293	4263	4252	4252
q2	321	220	224	220
q3	4135	4136	4162	4136
q4	2753	2755	2738	2738
q5	7330	7176	7262	7176
q6	234	119	115	115
q7	3233	2832	2813	2813
q8	4315	4408	4405	4405
q9	17078	16837	16981	16837
q10	4171	4228	4223	4223
q11	775	671	679	671
q12	1005	833	842	833
q13	7228	3745	3715	3715
q14	449	421	429	421
q15	492	454	453	453
q16	758	696	691	691
q17	3731	3791	3817	3791
q18	8754	8745	8736	8736
q19	1697	1691	1645	1645
q20	2378	2133	2117	2117
q21	8531	8296	8388	8296
q22	1045	994	938	938
Total cold run time: 84706 ms
Total hot run time: 79222 ms

doris-robot · 2024-02-05T07:30:39Z

TPC-DS: Total hot run time: 243079 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e3cb12e6c6e882df8abebcd5ddd678317aef40be, data reload: false

query1	916	383	379	379
query2	6540	2112	1981	1981
query3	6910	200	203	200
query4	20084	17908	17881	17881
query5	19724	6273	6243	6243
query6	280	227	237	227
query7	4156	291	300	291
query8	244	231	214	214
query9	3151	2716	2650	2650
query10	405	291	303	291
query11	11274	10713	10723	10713
query12	118	72	71	71
query13	5662	617	649	617
query14	17779	13194	13065	13065
query15	359	222	233	222
query16	6435	256	256	256
query17	3247	859	865	859
query18	2270	404	392	392
query19	209	138	146	138
query20	77	81	70	70
query21	188	93	91	91
query22	5336	5067	5112	5067
query23	32454	31697	31830	31697
query24	6901	6490	6408	6408
query25	527	415	407	407
query26	532	157	154	154
query27	1864	290	287	287
query28	6085	2250	2218	2218
query29	2805	2718	2799	2718
query30	238	158	157	157
query31	893	705	726	705
query32	62	57	59	57
query33	390	251	253	251
query34	855	475	486	475
query35	1147	896	952	896
query36	1624	1601	1831	1601
query37	89	61	59	59
query38	3182	3038	2938	2938
query39	1374	1329	1333	1329
query40	206	93	96	93
query41	36	36	32	32
query42	100	90	85	85
query43	670	674	601	601
query44	1169	717	718	717
query45	233	232	224	224
query46	1233	961	962	961
query47	1705	1934	1775	1775
query48	991	666	643	643
query49	614	357	357	357
query50	869	580	630	580
query51	5482	5423	5405	5405
query52	92	71	76	71
query53	438	306	304	304
query54	2416	2265	2270	2265
query55	83	77	79	77
query56	211	201	195	195
query57	1233	1180	1009	1009
query58	206	201	192	192
query59	3565	3104	3240	3104
query60	203	178	210	178
query61	81	83	79	79
query62	808	483	470	470
query63	472	321	321	321
query64	1998	1284	1329	1284
query65	3645	3604	3591	3591
query66	808	364	349	349
query67	15857	16108	15802	15802
query68	8903	646	663	646
query69	566	343	345	343
query70	1853	2050	1707	1707
query71	410	299	302	299
query72	4413	3250	3310	3250
query73	719	311	308	308
query74	6309	5921	5797	5797
query75	4765	3578	3442	3442
query76	5571	1175	1200	1175
query77	940	244	244	244
query78	32447	53256	55007	53256
query79	15604	636	647	636
query80	4610	369	370	369
query81	570	233	233	233
query82	1889	93	93	93
query83	334	130	129	129
query84	252	67	67	67
query85	1808	273	265	265
query86	446	432	384	384
query87	3224	2986	3009	2986
query88	7166	2299	2298	2298
query89	461	279	289	279
query90	2543	214	207	207
query91	156	112	127	112
query92	65	50	51	50
query93	6245	594	550	550
query94	1786	198	204	198
query95	1101	1062	1042	1042
query96	638	324	321	321
query97	6382	6309	6409	6309
query98	178	171	171	171
query99	3687	874	903	874
Total cold run time: 347851 ms
Total hot run time: 243079 ms

doris-robot · 2024-02-05T07:36:06Z

ClickBench: Total hot run time: 30.98 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e3cb12e6c6e882df8abebcd5ddd678317aef40be, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.25	0.05	0.04
query4	1.83	0.10	0.07
query5	0.53	0.53	0.52
query6	1.24	0.62	0.62
query7	0.02	0.02	0.01
query8	0.03	0.02	0.03
query9	0.51	0.50	0.48
query10	0.54	0.54	0.54
query11	0.11	0.08	0.08
query12	0.11	0.09	0.09
query13	0.62	0.61	0.61
query14	0.78	0.78	0.79
query15	0.77	0.77	0.77
query16	0.36	0.38	0.35
query17	1.00	1.00	1.00
query18	0.23	0.25	0.23
query19	1.82	1.82	1.85
query20	0.01	0.01	0.01
query21	15.47	0.56	0.56
query22	1.77	2.01	1.80
query23	17.43	0.93	0.92
query24	8.72	0.95	1.35
query25	1.58	0.11	0.10
query26	0.38	0.14	0.14
query27	0.11	0.11	0.11
query28	5.35	0.73	0.71
query29	12.61	2.28	2.29
query30	0.57	0.56	0.50
query31	2.82	0.37	0.38
query32	3.40	0.49	0.49
query33	3.07	3.08	3.08
query34	15.27	4.78	4.80
query35	4.86	4.86	4.83
query36	1.04	1.01	1.02
query37	0.05	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.66 s
Total hot run time: 30.98 s

doris-robot · 2024-02-05T07:39:19Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e3cb12e6c6e882df8abebcd5ddd678317aef40be with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.2 seconds inserted 10000000 Rows, about 471K ops/s

…0 (1day) (apache#30771) (apache#30798)

dataroaring approved these changes Feb 5, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 5, 2024

github-actions bot added the reviewed label Feb 5, 2024

dataroaring added 10 commits February 5, 2024 14:33

Update test_show_create_table_and_views.out

97bf989

Update test_show_create_table_and_views.out

6b0b33e

Update test_show_create_table_and_views.out

3fa4d89

Update test_show_create_table_and_views.out

e067dc7

Update test_show_create_table_and_views.out

dc94eea

Update test_show_create_table_and_views.out

da739e3

Update test_show_create_table_and_views.out

3919aaf

Update test_show_create_table_and_views.out

59677c6

Update test_show_create_table_and_views.out

43e8871

Update test_show_create_table_and_views.out

e3cb12e

dataroaring merged commit aa1d180 into apache:branch-2.0 Feb 5, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[improvement](binlog) Change BinlogConfig default TTL_SECONDS to 8640…

9b4783a

…0 (1day) (apache#30771) (apache#30798)

Conversation

JackDrogon commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

dataroaring commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

doris-robot commented Feb 4, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 5, 2024

Uh oh!

github-actions bot commented Feb 5, 2024

Uh oh!

dataroaring commented Feb 5, 2024

Uh oh!

doris-robot commented Feb 5, 2024

Uh oh!

doris-robot commented Feb 5, 2024

Uh oh!

doris-robot commented Feb 5, 2024

Uh oh!

doris-robot commented Feb 5, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants