[fix](fd) remove fd from cache when tablet is deleted by dataroaring · Pull Request #34004 · apache/doris

dataroaring · 2024-04-23T06:08:40Z

When tablet is set shutdown, a compaction on it may be running. So we should remove fds from cache when tablet's dir is deleted.

#34003

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

When tablet is set shutdown, a compaction on it may be running. So we should remove fds from cache when tablet's dir is deleted.

doris-robot · 2024-04-23T06:08:45Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

dataroaring · 2024-04-23T06:09:26Z

run buildall

github-actions · 2024-04-23T06:15:57Z

clang-tidy review says "All clean, LGTM! 👍"

dataroaring · 2024-04-23T06:31:19Z

run buildall

github-actions · 2024-04-23T06:33:28Z

clang-tidy review says "All clean, LGTM! 👍"

dataroaring · 2024-04-23T06:42:39Z

run buildall

github-actions · 2024-04-23T06:49:49Z

clang-tidy review says "All clean, LGTM! 👍"

dataroaring · 2024-04-23T06:57:17Z

run buildall

github-actions · 2024-04-23T07:04:19Z

clang-tidy review says "All clean, LGTM! 👍"

dataroaring · 2024-04-23T13:39:59Z

run buildall

dataroaring · 2024-04-23T13:41:38Z

run buildall

github-actions · 2024-04-23T13:46:55Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-04-23T13:49:11Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-04-24T02:22:18Z

clang-tidy review says "All clean, LGTM! 👍"

dataroaring · 2024-04-24T02:29:12Z

run buildall

github-actions · 2024-04-24T02:58:25Z

clang-tidy review says "All clean, LGTM! 👍"

dataroaring · 2024-04-24T03:51:34Z

run buildall

dataroaring · 2024-04-24T04:06:48Z

run buildall

dataroaring · 2024-04-24T04:13:26Z

run buildall

github-actions · 2024-04-24T04:14:36Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-04-24T04:20:50Z

clang-tidy review says "All clean, LGTM! 👍"

dataroaring · 2024-04-24T06:30:02Z

run buildall

github-actions · 2024-04-24T06:37:29Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-24T07:10:49Z

TPC-H: Total hot run time: 50187 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b5fff855be23c9b6e40059191bd1d97f8d3b5b9a, data reload: false

------ Round 1 ----------------------------------
q1	17883	4407	4397	4397
q2	2050	152	145	145
q3	10487	1866	1855	1855
q4	10316	1242	1308	1242
q5	8466	3964	4060	3964
q6	229	120	122	120
q7	2008	1567	1595	1567
q8	9298	2700	2703	2700
q9	10937	10659	10833	10659
q10	8689	3511	3471	3471
q11	425	233	248	233
q12	473	296	301	296
q13	18330	3974	3968	3968
q14	350	337	325	325
q15	508	457	465	457
q16	702	588	599	588
q17	1129	988	1006	988
q18	7288	6924	6864	6864
q19	1709	1565	1532	1532
q20	513	332	320	320
q21	4465	4113	4109	4109
q22	502	403	387	387
Total cold run time: 116757 ms
Total hot run time: 50187 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4331	4294	4315	4294
q2	316	221	222	221
q3	4138	4173	4093	4093
q4	2738	2742	2718	2718
q5	7292	7254	7247	7247
q6	238	120	118	118
q7	3283	2918	2782	2782
q8	4315	4433	4572	4433
q9	17508	17277	17268	17268
q10	4224	4249	4250	4249
q11	789	676	659	659
q12	1033	869	860	860
q13	7077	3701	3741	3701
q14	447	425	431	425
q15	501	450	458	450
q16	755	710	723	710
q17	3775	3800	3820	3800
q18	8822	8701	8717	8701
q19	1720	1687	1663	1663
q20	2409	2130	2154	2130
q21	8520	8524	8525	8524
q22	1018	971	953	953
Total cold run time: 85249 ms
Total hot run time: 79999 ms

doris-robot · 2024-04-24T07:13:33Z

TeamCity be ut coverage result:
Function Coverage: 37.80% (8069/21348)
Line Coverage: 29.46% (65901/223683)
Region Coverage: 28.94% (33923/117230)
Branch Coverage: 24.80% (17412/70212)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b5fff855be23c9b6e40059191bd1d97f8d3b5b9a_b5fff855be23c9b6e40059191bd1d97f8d3b5b9a/report/index.html

doris-robot · 2024-04-24T07:23:32Z

TPC-DS: Total hot run time: 202632 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b5fff855be23c9b6e40059191bd1d97f8d3b5b9a, data reload: false

query1	939	392	379	379
query2	6553	2735	2649	2649
query3	6915	195	198	195
query4	20370	17928	17963	17928
query5	19718	6535	6515	6515
query6	300	212	228	212
query7	4159	289	293	289
query8	247	279	238	238
query9	3139	2728	2650	2650
query10	412	281	301	281
query11	11294	10664	10664	10664
query12	114	71	69	69
query13	5576	687	660	660
query14	17523	13784	13623	13623
query15	347	232	245	232
query16	6464	264	252	252
query17	1731	1461	855	855
query18	2326	406	399	399
query19	202	146	151	146
query20	74	77	72	72
query21	185	92	93	92
query22	5178	5029	4939	4939
query23	32566	31892	31766	31766
query24	7014	6463	6438	6438
query25	520	410	422	410
query26	532	158	162	158
query27	1908	300	292	292
query28	6165	2398	2347	2347
query29	2893	2753	2786	2753
query30	236	164	162	162
query31	909	723	716	716
query32	65	59	58	58
query33	393	250	242	242
query34	838	455	474	455
query35	1124	989	866	866
query36	1617	1165	1337	1165
query37	92	60	58	58
query38	3107	2930	2893	2893
query39	1366	1321	1315	1315
query40	197	99	99	99
query41	37	40	34	34
query42	82	78	78	78
query43	750	768	722	722
query44	1117	731	726	726
query45	246	229	226	226
query46	1223	948	976	948
query47	1789	1852	1612	1612
query48	1006	719	722	719
query49	611	361	381	361
query50	870	618	574	574
query51	4772	4628	4662	4628
query52	89	71	85	71
query53	440	322	313	313
query54	2656	2450	2465	2450
query55	97	87	78	78
query56	211	204	192	192
query57	1301	1111	1146	1111
query58	220	191	188	188
query59	4007	4064	3943	3943
query60	209	198	192	192
query61	90	81	81	81
query62	815	470	515	470
query63	467	328	336	328
query64	2517	1548	1501	1501
query65	3648	3573	3572	3572
query66	779	367	384	367
query67	15499	14754	17314	14754
query68	9531	687	674	674
query69	561	332	341	332
query70	1810	1266	1297	1266
query71	402	292	299	292
query72	6509	3486	3440	3440
query73	728	321	323	321
query74	6296	5862	5880	5862
query75	5333	3796	3629	3629
query76	6092	1177	1208	1177
query77	1042	253	251	251
query78	12870	11907	11916	11907
query79	11009	647	656	647
query80	1141	400	386	386
query81	484	231	234	231
query82	1566	96	89	89
query83	157	132	131	131
query84	252	69	67	67
query85	858	303	306	303
query86	329	287	308	287
query87	3246	2991	2999	2991
query88	4936	2321	2330	2321
query89	459	304	293	293
query90	1959	204	203	203
query91	176	132	135	132
query92	58	50	53	50
query93	6593	570	572	570
query94	701	209	206	206
query95	1117	1088	1063	1063
query96	649	332	316	316
query97	6598	6379	6460	6379
query98	186	179	171	171
query99	3021	874	890	874
Total cold run time: 317525 ms
Total hot run time: 202632 ms

doris-robot · 2024-04-24T07:28:58Z

ClickBench: Total hot run time: 30.37 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b5fff855be23c9b6e40059191bd1d97f8d3b5b9a, data reload: false

query1	0.03	0.02	0.01
query2	0.06	0.02	0.02
query3	0.24	0.05	0.04
query4	1.80	0.06	0.06
query5	0.53	0.53	0.52
query6	1.25	0.62	0.61
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.47	0.48
query10	0.55	0.54	0.54
query11	0.13	0.08	0.09
query12	0.11	0.09	0.09
query13	0.62	0.61	0.62
query14	0.76	0.79	0.79
query15	0.79	0.76	0.76
query16	0.39	0.39	0.35
query17	1.02	1.01	0.99
query18	0.21	0.30	0.22
query19	1.93	1.83	1.86
query20	0.02	0.01	0.01
query21	15.48	0.55	0.56
query22	1.95	2.16	1.66
query23	16.95	0.97	0.92
query24	4.27	0.82	1.47
query25	0.33	0.10	0.05
query26	0.57	0.17	0.17
query27	0.05	0.04	0.03
query28	8.54	0.73	0.70
query29	12.63	2.31	2.14
query30	0.59	0.51	0.52
query31	2.82	0.38	0.37
query32	3.39	0.50	0.49
query33	3.10	3.10	3.04
query34	15.27	4.83	4.81
query35	4.90	4.84	4.82
query36	1.05	1.01	1.01
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 103.27 s
Total hot run time: 30.37 s

doris-robot · 2024-04-24T07:32:13Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b5fff855be23c9b6e40059191bd1d97f8d3b5b9a with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

github-actions · 2024-04-24T12:08:19Z

PR approved by anyone and no changes requested.

[fix](fd) remove fd from cache when tablet is deleted

7c4b84a

When tablet is set shutdown, a compaction on it may be running. So we should remove fds from cache when tablet's dir is deleted.

fix

5b3dbc4

fix

74d3803

fix

8b2d976

fix

a1051da

fix

5a04725

fix

9e76f22

dataroaring added 2 commits April 24, 2024 10:50

fix

26264dd

fix

167ef86

fix

29a06f0

fix

eb48d8b

fix

b5fff85

gavinchou approved these changes Apr 24, 2024

View reviewed changes

github-actions bot added the reviewed label Apr 24, 2024

dataroaring merged commit 422a92a into apache:branch-2.0 Apr 25, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[fix](fd) remove fd from cache when tablet is deleted (apache#34004)

d61e698

Conversation

dataroaring commented Apr 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 23, 2024

Uh oh!

dataroaring commented Apr 23, 2024

Uh oh!

github-actions bot commented Apr 23, 2024

Uh oh!

dataroaring commented Apr 23, 2024

Uh oh!

github-actions bot commented Apr 23, 2024

Uh oh!

dataroaring commented Apr 23, 2024

Uh oh!

github-actions bot commented Apr 23, 2024

Uh oh!

dataroaring commented Apr 23, 2024

Uh oh!

github-actions bot commented Apr 23, 2024

Uh oh!

dataroaring commented Apr 23, 2024

Uh oh!

dataroaring commented Apr 23, 2024

Uh oh!

github-actions bot commented Apr 23, 2024

Uh oh!

github-actions bot commented Apr 23, 2024

Uh oh!

github-actions bot commented Apr 24, 2024

Uh oh!

dataroaring commented Apr 24, 2024

Uh oh!

github-actions bot commented Apr 24, 2024

Uh oh!

dataroaring commented Apr 24, 2024

Uh oh!

dataroaring commented Apr 24, 2024

Uh oh!

dataroaring commented Apr 24, 2024

Uh oh!

github-actions bot commented Apr 24, 2024

Uh oh!

github-actions bot commented Apr 24, 2024

Uh oh!

dataroaring commented Apr 24, 2024

Uh oh!

github-actions bot commented Apr 24, 2024

Uh oh!

doris-robot commented Apr 24, 2024

Uh oh!

doris-robot commented Apr 24, 2024

Uh oh!

doris-robot commented Apr 24, 2024

Uh oh!

doris-robot commented Apr 24, 2024

Uh oh!

doris-robot commented Apr 24, 2024

Uh oh!

github-actions bot commented Apr 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dataroaring commented Apr 23, 2024 •

edited

Loading