Skip to content

[fix](move-memtable) check load timeout before close wait#30526

Merged
dataroaring merged 1 commit intoapache:masterfrom
kaijchen:fix-negative-timeout
Jan 30, 2024
Merged

[fix](move-memtable) check load timeout before close wait#30526
dataroaring merged 1 commit intoapache:masterfrom
kaijchen:fix-negative-timeout

Conversation

@kaijchen
Copy link
Member

Proposed changes

close wait timeout = load timeout - elapsed time

If remaining time is not positive, return timeout

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@kaijchen
Copy link
Member Author

run buildall

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 29, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38074 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 24a4b64e44c0b6cc26230c853a57fc61b100ee1e, data reload: false

------ Round 1 ----------------------------------
q1	17752	7385	5536	5536
q2	2055	140	131	131
q3	9812	936	909	909
q4	4726	728	708	708
q5	7646	2913	2749	2749
q6	186	120	119	119
q7	1133	729	711	711
q8	9341	2000	2018	2000
q9	7217	6317	6284	6284
q10	8061	2405	2413	2405
q11	422	207	208	207
q12	752	280	286	280
q13	18017	3315	3289	3289
q14	275	243	253	243
q15	525	499	514	499
q16	487	407	421	407
q17	949	574	543	543
q18	6980	6076	5994	5994
q19	1578	1380	1345	1345
q20	614	361	347	347
q21	7147	3206	3079	3079
q22	806	315	289	289
Total cold run time: 106481 ms
Total hot run time: 38074 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5179	5057	4851	4851
q2	338	244	237	237
q3	2966	2666	2673	2666
q4	1861	1652	1548	1548
q5	5356	5282	5358	5282
q6	205	117	118	117
q7	2110	1797	1763	1763
q8	3136	3210	3266	3210
q9	8526	8593	16167	8593
q10	8431	3426	3427	3426
q11	556	489	463	463
q12	797	641	591	591
q13	16945	3093	3058	3058
q14	288	251	255	251
q15	539	489	487	487
q16	560	463	476	463
q17	2107	1583	1716	1583
q18	8905	8518	8141	8141
q19	23150	1607	1518	1518
q20	4898	1937	1916	1916
q21	14522	4907	4882	4882
q22	1053	482	486	482
Total cold run time: 112428 ms
Total hot run time: 55528 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.19% (8621/23823)
Line Coverage: 28.25% (70519/249585)
Region Coverage: 27.25% (36382/133524)
Branch Coverage: 24.05% (18649/77538)
Coverage Report: http://coverage.selectdb-in.cc/coverage/24a4b64e44c0b6cc26230c853a57fc61b100ee1e_24a4b64e44c0b6cc26230c853a57fc61b100ee1e/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 173944 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 24a4b64e44c0b6cc26230c853a57fc61b100ee1e, data reload: false

query1	960	338	326	326
query2	6545	2114	2050	2050
query3	6696	202	194	194
query4	33954	22627	22629	22627
query5	4750	403	361	361
query6	600	183	178	178
query7	5887	279	258	258
query8	697	189	207	189
query9	10034	2306	2276	2276
query10	415	210	207	207
query11	17898	15478	15340	15340
query12	115	69	65	65
query13	1671	372	400	372
query14	9173	6454	6553	6454
query15	209	173	181	173
query16	5705	273	249	249
query17	936	477	468	468
query18	1805	269	251	251
query19	187	134	132	132
query20	71	71	67	67
query21	204	131	131	131
query22	4903	4660	4691	4660
query23	31366	30510	30406	30406
query24	11672	2771	2765	2765
query25	567	305	310	305
query26	892	140	136	136
query27	2972	278	278	278
query28	7090	1826	1818	1818
query29	985	601	635	601
query30	284	134	143	134
query31	876	710	715	710
query32	81	49	50	49
query33	702	217	208	208
query34	1133	462	464	462
query35	853	762	767	762
query36	1300	1194	1243	1194
query37	90	61	59	59
query38	3284	3136	3174	3136
query39	1335	1250	1238	1238
query40	208	96	89	89
query41	37	33	36	33
query42	94	76	87	76
query43	537	487	500	487
query44	1103	696	683	683
query45	189	181	176	176
query46	1044	674	656	656
query47	1655	1541	1483	1483
query48	385	327	308	308
query49	1102	284	296	284
query50	683	306	308	306
query51	5260	5135	5143	5135
query52	92	81	73	73
query53	325	259	254	254
query54	249	179	183	179
query55	79	79	71	71
query56	176	164	160	160
query57	946	864	887	864
query58	181	158	161	158
query59	2895	2571	2535	2535
query60	201	179	177	177
query61	83	84	85	84
query62	676	353	345	345
query63	285	264	248	248
query64	4584	1775	1741	1741
query65	3345	3235	3228	3228
query66	1139	320	331	320
query67	14401	14164	14318	14164
query68	4117	466	491	466
query69	452	310	307	307
query70	1546	1511	1456	1456
query71	294	217	217	217
query72	4028	2848	2863	2848
query73	689	320	309	309
query74	6642	6372	6280	6280
query75	2826	2295	2275	2275
query76	3166	1038	1021	1021
query77	396	229	235	229
query78	9136	8715	8495	8495
query79	2742	491	483	483
query80	2054	324	317	317
query81	527	194	199	194
query82	700	86	83	83
query83	239	126	120	120
query84	286	74	77	74
query85	2187	354	318	318
query86	504	398	405	398
query87	3425	3267	3287	3267
query88	3813	2165	2189	2165
query89	406	361	366	361
query90	1915	195	183	183
query91	155	117	122	117
query92	55	43	44	43
query93	2314	448	403	403
query94	1361	161	158	158
query95	492	460	450	450
query96	611	320	308	308
query97	4272	4115	4107	4107
query98	214	191	187	187
query99	1011	703	699	699
Total cold run time: 283426 ms
Total hot run time: 173944 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.72 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 24a4b64e44c0b6cc26230c853a57fc61b100ee1e, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.02	0.02
query3	0.23	0.06	0.05
query4	1.68	0.09	0.09
query5	0.52	0.51	0.51
query6	1.17	0.64	0.61
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.57	0.50	0.50
query10	0.56	0.55	0.55
query11	0.12	0.09	0.08
query12	0.11	0.09	0.10
query13	0.61	0.60	0.60
query14	0.78	0.80	0.80
query15	0.80	0.77	0.77
query16	0.36	0.40	0.39
query17	1.02	1.03	1.01
query18	0.25	0.22	0.24
query19	1.86	1.80	1.79
query20	0.02	0.01	0.01
query21	15.40	0.57	0.55
query22	2.33	2.41	2.00
query23	16.74	0.84	0.81
query24	2.60	1.15	0.45
query25	0.29	0.20	0.14
query26	0.49	0.16	0.15
query27	0.05	0.05	0.07
query28	12.04	0.84	0.83
query29	12.61	3.21	3.25
query30	0.59	0.62	0.55
query31	2.79	0.34	0.34
query32	3.35	0.48	0.48
query33	3.26	3.23	3.21
query34	15.82	4.29	4.25
query35	4.31	4.28	4.26
query36	1.10	1.05	1.08
query37	0.07	0.05	0.05
query38	0.03	0.03	0.03
query39	0.03	0.01	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.01
Total cold run time: 105 s
Total hot run time: 30.72 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 24a4b64e44c0b6cc26230c853a57fc61b100ee1e with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

_timeout_watch.elapsed_time() / 1000 / 1000;
if (remain_ms <= 0) {
LOG(WARNING) << "load timed out before close waiting, load_id="
<< print_id(_load_id);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better print remain_ms too

@dataroaring dataroaring merged commit d848c81 into apache:master Jan 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.0 reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants