Skip to content

Comments

[fix](dns-cache) do not detach the refresh thread#33182

Merged
xiaokang merged 2 commits intoapache:masterfrom
morningman:dns_cache_bug
Apr 3, 2024
Merged

[fix](dns-cache) do not detach the refresh thread#33182
xiaokang merged 2 commits intoapache:masterfrom
morningman:dns_cache_bug

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Apr 2, 2024 •
edited
Loading

Proposed changes

  1. do not detach the refresh thread, or it can not be joined.
  2. Released the dns cache at last when destroying the exec env, avoid other thread still using it.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions
Copy link
Contributor

github-actions bot commented Apr 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@github-actions
Copy link
Contributor

github-actions bot commented Apr 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

zhiqiang-hhhh
zhiqiang-hhhh approved these changes Apr 2, 2024
View reviewed changes
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman
Copy link
Contributor Author

morningman commented Apr 2, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Apr 2, 2024

PR approved by anyone and no changes requested.

morningman added a commit to morningman/doris that referenced this pull request Apr 2, 2024
@morningman
[fix](dns-cache) do not detach the refresh thread apache#33182
264aeca
@morningman morningman mentioned this pull request Apr 2, 2024
Merged
@doris-robot
Copy link

doris-robot commented Apr 2, 2024

TeamCity be ut coverage result:
Function Coverage: 35.65% (8880/24912)
Line Coverage: 27.37% (72904/266330)
Region Coverage: 26.55% (37706/142004)
Branch Coverage: 23.34% (19212/82310)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f3047b498818ce4403d554912269e2e0ce833819_f3047b498818ce4403d554912269e2e0ce833819/report/index.html

@doris-robot
Copy link

doris-robot commented Apr 2, 2024

TPC-H: Total hot run time: 39004 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f3047b498818ce4403d554912269e2e0ce833819, data reload: false

------ Round 1 ----------------------------------
q1	17701	4099	4182	4099
q2	2322	195	192	192
q3	11732	1340	1451	1340
q4	10702	889	1075	889
q5	8483	3036	2983	2983
q6	222	136	135	135
q7	1113	627	622	622
q8	9408	2056	2034	2034
q9	6768	6182	6129	6129
q10	8416	3517	3505	3505
q11	416	251	245	245
q12	383	215	211	211
q13	17770	2895	2945	2895
q14	279	237	254	237
q15	530	489	464	464
q16	492	389	376	376
q17	968	911	897	897
q18	7293	6620	6486	6486
q19	1619	1548	1555	1548
q20	621	336	298	298
q21	3516	3124	3116	3116
q22	362	303	313	303
Total cold run time: 111116 ms
Total hot run time: 39004 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4060	4081	4054	4054
q2	328	218	217	217
q3	2967	2950	2936	2936
q4	1879	1842	1801	1801
q5	5198	5220	5241	5220
q6	208	123	121	121
q7	2230	1799	1807	1799
q8	3218	3277	3291	3277
q9	8483	8419	8434	8419
q10	3753	3808	3826	3808
q11	540	450	448	448
q12	710	557	558	557
q13	6625	2896	2908	2896
q14	287	251	262	251
q15	507	485	473	473
q16	459	408	400	400
q17	1730	1698	1682	1682
q18	7566	7308	7165	7165
q19	1642	1634	1632	1632
q20	1940	1708	1718	1708
q21	4945	4721	4687	4687
q22	487	421	434	421
Total cold run time: 59762 ms
Total hot run time: 53972 ms

@doris-robot
Copy link

doris-robot commented Apr 2, 2024

TPC-DS: Total hot run time: 180355 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f3047b498818ce4403d554912269e2e0ce833819, data reload: false

query1	1230	1124	367	367
query2	6477	1957	1772	1772
query3	6658	224	233	224
query4	24234	21444	21500	21444
query5	4278	401	400	400
query6	277	184	181	181
query7	4606	308	299	299
query8	226	174	172	172
query9	8451	2266	2264	2264
query10	573	257	266	257
query11	14813	14527	14487	14487
query12	147	104	97	97
query13	1650	394	385	385
query14	8644	6893	6765	6765
query15	221	188	174	174
query16	7162	285	278	278
query17	985	598	580	580
query18	1905	289	283	283
query19	202	164	162	162
query20	98	94	89	89
query21	198	125	130	125
query22	5062	4852	4811	4811
query23	33437	32707	32670	32670
query24	12383	3108	3126	3108
query25	683	386	405	386
query26	1898	163	159	159
query27	3005	319	330	319
query28	6795	1862	1825	1825
query29	1335	583	583	583
query30	308	171	168	168
query31	1016	732	725	725
query32	98	60	56	56
query33	704	248	253	248
query34	1058	479	484	479
query35	851	702	702	702
query36	986	878	864	864
query37	288	75	76	75
query38	3514	3372	3402	3372
query39	1547	1523	1524	1523
query40	304	130	136	130
query41	46	46	45	45
query42	110	101	104	101
query43	447	406	407	406
query44	1073	702	699	699
query45	280	263	248	248
query46	1065	786	772	772
query47	1901	1807	1806	1806
query48	382	308	304	304
query49	1166	361	361	361
query50	798	388	400	388
query51	6868	6776	6673	6673
query52	109	100	104	100
query53	362	289	285	285
query54	325	238	246	238
query55	87	84	79	79
query56	245	227	233	227
query57	1227	1127	1098	1098
query58	256	224	228	224
query59	2476	2298	2368	2298
query60	258	239	248	239
query61	111	109	113	109
query62	705	464	457	457
query63	309	281	282	281
query64	6439	3462	3232	3232
query65	3071	3009	3016	3009
query66	1440	336	326	326
query67	15617	14820	15047	14820
query68	5046	566	576	566
query69	490	335	335	335
query70	1150	1114	1138	1114
query71	412	280	279	279
query72	6331	2715	2463	2463
query73	718	325	323	323
query74	6869	6357	6347	6347
query75	3007	2282	2267	2267
query76	3255	1100	1205	1100
query77	386	260	247	247
query78	10929	10176	10060	10060
query79	8305	533	518	518
query80	1949	419	431	419
query81	531	247	233	233
query82	1586	102	101	101
query83	271	162	163	162
query84	276	88	90	88
query85	2144	302	283	283
query86	479	271	284	271
query87	3689	3514	3491	3491
query88	4323	2321	2329	2321
query89	497	367	373	367
query90	1970	178	182	178
query91	135	106	105	105
query92	63	52	56	52
query93	6319	523	531	523
query94	1274	195	193	193
query95	438	346	332	332
query96	608	269	271	269
query97	2716	2509	2488	2488
query98	227	219	209	209
query99	1264	853	818	818
Total cold run time: 293137 ms
Total hot run time: 180355 ms

@doris-robot
Copy link

doris-robot commented Apr 2, 2024

ClickBench: Total hot run time: 30.13 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f3047b498818ce4403d554912269e2e0ce833819, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.04	0.05
query4	1.68	0.07	0.07
query5	0.49	0.48	0.49
query6	1.12	0.65	0.66
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.51	0.51
query10	0.56	0.56	0.56
query11	0.14	0.11	0.11
query12	0.13	0.11	0.12
query13	0.62	0.61	0.59
query14	0.77	0.78	0.79
query15	0.85	0.84	0.83
query16	0.36	0.36	0.35
query17	0.98	0.96	0.96
query18	0.25	0.25	0.24
query19	1.82	1.70	1.72
query20	0.02	0.01	0.01
query21	15.45	0.77	0.72
query22	3.09	5.01	2.04
query23	17.67	1.52	1.12
query24	1.47	0.20	0.22
query25	0.12	0.09	0.09
query26	0.28	0.17	0.18
query27	0.09	0.08	0.08
query28	13.84	0.95	0.93
query29	12.76	3.28	3.20
query30	0.25	0.06	0.05
query31	2.85	0.39	0.38
query32	3.28	0.48	0.47
query33	2.83	2.82	2.83
query34	15.47	4.33	4.31
query35	4.38	4.38	4.34
query36	0.67	0.47	0.47
query37	0.21	0.16	0.19
query38	0.16	0.17	0.16
query39	0.05	0.04	0.04
query40	0.20	0.14	0.16
query41	0.10	0.04	0.05
query42	0.07	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 106.1 s
Total hot run time: 30.13 s

@doris-robot
Copy link

doris-robot commented Apr 2, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f3047b498818ce4403d554912269e2e0ce833819 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.3 seconds inserted 10000000 Rows, about 613K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 3, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Apr 3, 2024

PR approved by at least one committer and no changes requested.

xiaokang pushed a commit that referenced this pull request Apr 3, 2024
@morningman
[fix](dns-cache) do not detach the refresh thread #33182 (#33184)
fbf9f62
@xiaokang xiaokang merged commit 8dac29c into apache:master Apr 3, 2024
morningman added a commit to morningman/doris that referenced this pull request Apr 7, 2024
@morningman
[fix](dns-cache) do not detach the refresh thread (apache#33182)
4669729
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@morningman
[fix](dns-cache) do not detach the refresh thread apache#33182 (apach…
bfc8383 
…e#33184)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

+1 more reviewer

@zhiqiang-hhhh zhiqiang-hhhh zhiqiang-hhhh approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.8-merged dev/2.1.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@morningman @doris-robot @yiguolei @zhiqiang-hhhh @xiaokang