Skip to content

[enhance](InternalTable) Set timeout for cache hotspot table's insert stmt#36279

Merged
dataroaring merged 4 commits intoapache:masterfrom
ByteYue:set_timeout_for_cache_hotspot_table
Jun 15, 2024
Merged

[enhance](InternalTable) Set timeout for cache hotspot table's insert stmt#36279
dataroaring merged 4 commits intoapache:masterfrom
ByteYue:set_timeout_for_cache_hotspot_table

Conversation

@ByteYue
Copy link
Contributor

@ByteYue ByteYue commented Jun 13, 2024

Proposed changes

Previously the insert stmt of cache hot spot table doesn't set the insert timeout, sometimes it would last a long time. This pr adds one session variable to control it.

@doris-robot
Copy link

doris-robot commented Jun 13, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@ByteYue
Copy link
Contributor Author

ByteYue commented Jun 13, 2024

run buildall

gavinchou
gavinchou reviewed Jun 13, 2024
View reviewed changes
@ByteYue
Copy link
Contributor Author

ByteYue commented Jun 13, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 13, 2024

TPC-H: Total hot run time: 39884 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cef09159df3655c0232c808e9776203a375595ee, data reload: false

------ Round 1 ----------------------------------
q1	18101	6091	4267	4267
q2	2046	200	191	191
q3	10732	1083	1107	1083
q4	10795	843	808	808
q5	7545	2710	2659	2659
q6	227	145	138	138
q7	968	627	624	624
q8	9290	2068	2100	2068
q9	9280	6605	6576	6576
q10	9127	3665	3707	3665
q11	436	233	238	233
q12	453	228	226	226
q13	17762	2963	3024	2963
q14	267	214	221	214
q15	515	470	470	470
q16	506	370	385	370
q17	966	674	724	674
q18	7935	7426	7487	7426
q19	2473	1452	1518	1452
q20	669	301	316	301
q21	4908	3150	3139	3139
q22	372	337	338	337
Total cold run time: 115373 ms
Total hot run time: 39884 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4361	4242	4261	4242
q2	358	259	259	259
q3	2946	2702	2743	2702
q4	1890	1601	1639	1601
q5	5246	5259	5280	5259
q6	212	124	131	124
q7	2132	1762	1735	1735
q8	3175	3294	3284	3284
q9	8274	8322	8322	8322
q10	3877	3636	3616	3616
q11	616	484	480	480
q12	742	585	605	585
q13	16295	2948	2978	2948
q14	285	252	259	252
q15	525	482	476	476
q16	459	398	400	398
q17	1765	1482	1488	1482
q18	7648	7636	7476	7476
q19	3230	1487	1477	1477
q20	1988	1776	1752	1752
q21	4900	4665	4799	4665
q22	602	552	547	547
Total cold run time: 71526 ms
Total hot run time: 53682 ms

@doris-robot
Copy link

doris-robot commented Jun 13, 2024

TPC-DS: Total hot run time: 172952 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cef09159df3655c0232c808e9776203a375595ee, data reload: false

query1	936	385	370	370
query2	6486	2453	2369	2369
query3	6651	208	205	205
query4	20193	17457	17330	17330
query5	4138	498	456	456
query6	246	159	159	159
query7	4590	307	296	296
query8	309	279	300	279
query9	8688	2455	2424	2424
query10	617	305	294	294
query11	10722	10108	10034	10034
query12	135	86	84	84
query13	1639	388	370	370
query14	9328	7481	7317	7317
query15	247	189	185	185
query16	8023	270	265	265
query17	1916	536	540	536
query18	2052	272	268	268
query19	198	150	152	150
query20	89	83	82	82
query21	211	130	131	130
query22	4274	4086	4112	4086
query23	33467	32935	33108	32935
query24	12025	2854	2770	2770
query25	654	356	361	356
query26	1744	157	154	154
query27	2970	319	314	314
query28	7620	2051	2035	2035
query29	1034	615	616	615
query30	288	149	149	149
query31	928	723	741	723
query32	93	52	56	52
query33	778	287	293	287
query34	970	477	478	477
query35	734	597	595	595
query36	1066	942	925	925
query37	167	71	71	71
query38	2896	2748	2779	2748
query39	833	791	789	789
query40	278	125	124	124
query41	56	52	53	52
query42	123	110	107	107
query43	574	577	553	553
query44	1187	711	739	711
query45	202	167	166	166
query46	1077	718	732	718
query47	1880	1781	1766	1766
query48	381	290	296	290
query49	1172	408	396	396
query50	756	392	384	384
query51	6859	6761	6583	6583
query52	98	96	90	90
query53	356	287	280	280
query54	998	465	448	448
query55	72	76	73	73
query56	278	265	261	261
query57	1133	1055	1043	1043
query58	245	231	250	231
query59	3451	2992	3121	2992
query60	294	271	295	271
query61	96	91	91	91
query62	644	437	439	437
query63	321	294	286	286
query64	9823	2198	1704	1704
query65	3187	3095	3112	3095
query66	1353	330	356	330
query67	15466	15021	14835	14835
query68	4570	543	531	531
query69	456	302	295	295
query70	1109	1137	1083	1083
query71	397	283	318	283
query72	7047	5807	5447	5447
query73	742	328	336	328
query74	6070	5561	5503	5503
query75	3566	2669	2642	2642
query76	2787	939	958	939
query77	432	308	301	301
query78	10179	9932	9892	9892
query79	2288	512	500	500
query80	984	477	462	462
query81	576	221	218	218
query82	772	108	103	103
query83	234	170	171	170
query84	231	82	83	82
query85	2026	285	341	285
query86	488	307	329	307
query87	3260	3125	3108	3108
query88	3965	2436	2421	2421
query89	464	395	379	379
query90	1818	199	201	199
query91	126	101	98	98
query92	58	52	52	52
query93	2419	522	506	506
query94	1277	194	190	190
query95	404	313	318	313
query96	580	280	275	275
query97	3194	3055	3011	3011
query98	219	201	193	193
query99	1277	835	864	835
Total cold run time: 276330 ms
Total hot run time: 172952 ms

@doris-robot
Copy link

doris-robot commented Jun 13, 2024

ClickBench: Total hot run time: 30.16 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cef09159df3655c0232c808e9776203a375595ee, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.04
query4	1.67	0.08	0.09
query5	0.50	0.49	0.49
query6	1.13	0.72	0.71
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.54	0.48	0.48
query10	0.54	0.55	0.54
query11	0.15	0.12	0.11
query12	0.14	0.12	0.11
query13	0.59	0.59	0.61
query14	0.78	0.79	0.78
query15	0.82	0.81	0.82
query16	0.36	0.35	0.36
query17	0.95	1.01	0.97
query18	0.21	0.24	0.24
query19	1.79	1.73	1.69
query20	0.02	0.01	0.01
query21	15.43	0.65	0.65
query22	4.02	7.80	1.72
query23	18.27	1.44	1.28
query24	2.17	0.20	0.23
query25	0.15	0.07	0.08
query26	0.27	0.18	0.17
query27	0.08	0.08	0.08
query28	13.21	1.02	0.99
query29	12.64	3.24	3.22
query30	0.25	0.07	0.06
query31	2.85	0.39	0.38
query32	3.28	0.47	0.47
query33	2.83	2.84	2.90
query34	17.22	4.41	4.40
query35	4.48	4.48	4.51
query36	0.64	0.45	0.46
query37	0.18	0.16	0.15
query38	0.14	0.14	0.14
query39	0.04	0.03	0.04
query40	0.18	0.14	0.14
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.12 s
Total hot run time: 30.16 s

@deardeng
Copy link
Contributor

deardeng commented Jun 14, 2024

内部表，统计数据的，插入是不是也可以，超时时间设置短点？

@ByteYue
Copy link
Contributor Author

ByteYue commented Jun 14, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 14, 2024

TPC-H: Total hot run time: 39554 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cef09159df3655c0232c808e9776203a375595ee, data reload: false

------ Round 1 ----------------------------------
q1	17638	4298	4293	4293
q2	2024	189	200	189
q3	10459	1119	983	983
q4	10187	796	825	796
q5	7503	2673	2630	2630
q6	218	134	136	134
q7	966	622	598	598
q8	9218	2056	2046	2046
q9	8801	6469	6434	6434
q10	8973	3697	3673	3673
q11	455	235	244	235
q12	449	230	236	230
q13	17771	2986	2972	2972
q14	270	229	229	229
q15	522	483	479	479
q16	527	373	399	373
q17	954	677	697	677
q18	8052	7446	7362	7362
q19	6666	1394	1441	1394
q20	659	316	318	316
q21	4923	3162	3852	3162
q22	395	349	349	349
Total cold run time: 117630 ms
Total hot run time: 39554 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4358	4209	4201	4201
q2	364	256	264	256
q3	2947	2916	2927	2916
q4	1976	1675	1679	1675
q5	5616	5509	5466	5466
q6	221	130	131	130
q7	2208	1901	1842	1842
q8	3269	3381	3414	3381
q9	8744	8743	8803	8743
q10	4083	3881	3700	3700
q11	598	485	492	485
q12	842	645	608	608
q13	16417	3146	3106	3106
q14	307	281	296	281
q15	515	486	480	480
q16	514	448	435	435
q17	1789	1513	1495	1495
q18	8080	7866	7787	7787
q19	1778	1694	1666	1666
q20	3139	1898	1870	1870
q21	6756	5038	4831	4831
q22	617	556	556	556
Total cold run time: 75138 ms
Total hot run time: 55910 ms

@doris-robot
Copy link

doris-robot commented Jun 14, 2024

TPC-DS: Total hot run time: 171740 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cef09159df3655c0232c808e9776203a375595ee, data reload: false

query1	948	382	387	382
query2	6473	2390	2347	2347
query3	6646	205	215	205
query4	19856	17407	17074	17074
query5	3601	476	475	475
query6	237	166	184	166
query7	4588	301	293	293
query8	309	296	282	282
query9	8527	2435	2394	2394
query10	572	327	297	297
query11	10559	9954	9956	9954
query12	118	95	88	88
query13	1673	368	366	366
query14	9842	7847	7789	7789
query15	230	195	185	185
query16	7707	266	265	265
query17	1908	533	528	528
query18	1772	268	271	268
query19	195	151	158	151
query20	94	85	80	80
query21	202	134	128	128
query22	4352	4073	3950	3950
query23	33728	34026	33790	33790
query24	10807	2913	2880	2880
query25	612	386	390	386
query26	719	167	157	157
query27	2222	318	335	318
query28	5937	2125	2131	2125
query29	887	665	660	660
query30	253	161	161	161
query31	965	784	765	765
query32	96	55	57	55
query33	698	312	280	280
query34	896	488	487	487
query35	756	662	641	641
query36	1144	992	999	992
query37	134	72	72	72
query38	2963	2825	2830	2825
query39	887	827	851	827
query40	217	139	129	129
query41	55	54	55	54
query42	113	106	102	102
query43	605	536	546	536
query44	1156	725	736	725
query45	191	161	166	161
query46	1065	718	722	718
query47	1837	1773	1778	1773
query48	384	289	297	289
query49	840	402	409	402
query50	758	391	399	391
query51	6776	6652	6723	6652
query52	109	88	102	88
query53	361	292	287	287
query54	876	439	437	437
query55	72	73	71	71
query56	276	263	259	259
query57	1125	1045	1040	1040
query58	242	240	249	240
query59	3463	3166	3118	3118
query60	289	278	279	278
query61	92	93	89	89
query62	615	420	453	420
query63	311	302	288	288
query64	8632	2222	1731	1731
query65	3170	3086	3122	3086
query66	746	330	324	324
query67	15364	14868	15669	14868
query68	4477	533	537	533
query69	521	443	405	405
query70	1209	1156	1097	1097
query71	451	288	274	274
query72	7148	5248	2747	2747
query73	738	323	317	317
query74	5793	5539	5473	5473
query75	3452	2676	2693	2676
query76	2944	967	901	901
query77	617	299	302	299
query78	10311	9924	9782	9782
query79	2270	533	529	529
query80	867	481	469	469
query81	555	216	223	216
query82	1209	102	107	102
query83	259	165	164	164
query84	231	86	87	86
query85	1355	278	268	268
query86	464	308	345	308
query87	3275	3076	3063	3063
query88	4144	2362	2351	2351
query89	462	395	393	393
query90	1752	191	186	186
query91	148	101	102	101
query92	58	49	52	49
query93	2204	500	496	496
query94	1082	186	181	181
query95	406	318	335	318
query96	587	269	267	267
query97	3188	3025	3052	3025
query98	291	204	206	204
query99	1233	834	816	816
Total cold run time: 267554 ms
Total hot run time: 171740 ms

@doris-robot
Copy link

doris-robot commented Jun 14, 2024

ClickBench: Total hot run time: 31.41 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cef09159df3655c0232c808e9776203a375595ee, data reload: false

query1	0.04	0.04	0.02
query2	0.08	0.04	0.04
query3	0.22	0.04	0.06
query4	1.70	0.09	0.09
query5	0.51	0.49	0.50
query6	1.13	0.72	0.74
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.50	0.47
query10	0.55	0.55	0.54
query11	0.15	0.12	0.12
query12	0.16	0.12	0.12
query13	0.60	0.59	0.61
query14	0.77	0.80	0.77
query15	0.84	0.83	0.80
query16	0.37	0.37	0.35
query17	0.98	0.98	1.07
query18	0.23	0.24	0.25
query19	1.79	1.79	1.83
query20	0.02	0.01	0.01
query21	15.41	0.66	0.65
query22	3.92	6.51	2.80
query23	18.28	1.47	1.21
query24	2.08	0.22	0.21
query25	0.15	0.10	0.09
query26	0.25	0.17	0.17
query27	0.07	0.08	0.08
query28	13.27	1.02	0.99
query29	12.65	3.32	3.30
query30	0.25	0.06	0.06
query31	2.86	0.40	0.37
query32	3.28	0.48	0.48
query33	2.89	2.96	2.89
query34	17.10	4.41	4.42
query35	4.45	4.40	4.48
query36	0.66	0.48	0.48
query37	0.19	0.16	0.15
query38	0.16	0.14	0.14
query39	0.05	0.03	0.04
query40	0.16	0.14	0.14
query41	0.10	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.08 s
Total hot run time: 31.41 s

@ByteYue
Copy link
Contributor Author

ByteYue commented Jun 14, 2024

内部表，统计数据的，插入是不是也可以，超时时间设置短点？

统计信息表已经有一个session variable可以控制的

dataroaring
dataroaring approved these changes Jun 15, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 15, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 15, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 15, 2024

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 551a2c0 into apache:master Jun 15, 2024
dataroaring pushed a commit that referenced this pull request Jun 21, 2024
@ByteYue @dataroaring
[enhance](InternalTable) Set timeout for cache hotspot table's insert…
0b66c23 
… stmt (#36279)

Previously the insert stmt of cache hot spot table doesn't set the
insert timeout, sometimes it would last a long time. This pr adds one
session variable to control it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ByteYue @doris-robot @deardeng @gavinchou @dataroaring