Skip to content

[fix](memory) Revert "[opt](memory) Refactor memory maintenance thread (#40344)"#40545

Merged
hello-stephen merged 1 commit intoapache:masterfrom
xinyiZzz:20240909_fix_memory2
Sep 9, 2024
Merged

[fix](memory) Revert "[opt](memory) Refactor memory maintenance thread (#40344)"#40545
hello-stephen merged 1 commit intoapache:masterfrom
xinyiZzz:20240909_fix_memory2

Conversation

@xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Sep 9, 2024 •
edited
Loading

This reverts commit 84ce945.

SIGABRT unknown detail explain (@0x4220) received by PID 16928 (TID 18624 OR 0x7fa0a663e700) from PID 16928; stack trace: ***
15:21:03    0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:421
15:21:03    1# 0x00007FA7C6131090 in /lib/x86_64-linux-gnu/libc.so.6
15:21:03    2# raise at ../sysdeps/unix/sysv/linux/raise.c:51
15:21:03    3# abort at /build/glibc-SzIz7B/glibc-2.31/stdlib/abort.c:81
15:21:03    4# 0x0000558ACC0918AD in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
15:21:03    5# google::LogMessage::SendToLog() in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
15:21:03    6# google::LogMessage::Flush() in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
15:21:03    7# google::LogMessageFatal::~LogMessageFatal() in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
15:21:03    8# doris::ThreadContext::consume_memory(long) const at /root/doris/be/src/runtime/thread_context.h:244
15:21:03    9# Allocator<false, false, false, DefaultMemoryAllocator>::release_memory(unsigned long) const in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
15:21:03   10# doris::vectorized::ColumnVector<unsigned char>::~ColumnVector() at /root/doris/be/src/vec/columns/column_vector.h:131
15:21:03   11# doris::vectorized::ColumnNullable::~ColumnNullable() at /root/doris/be/src/vec/columns/column_nullable.h:62
15:21:03   12# doris::vectorized::Block::~Block() at /root/doris/be/src/vec/core/block.h:92
15:21:03   13# std::vector<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >, std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > > >::~vector() at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:680
15:21:03   14# doris::Reusable::~Reusable() at /root/doris/be/src/service/point_query_executor.cpp:65
15:21:03   15# std::Sp_counted_base<(_gnu_cxx::_Lock_policy)2>::_M_release() at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:180
15:21:03   16# doris::LookupConnectionCache::CacheValue::~CacheValue() at /root/doris/be/src/service/point_query_executor.h:266
15:21:03   17# doris::LRUCache::set_capacity(unsigned long) at /root/doris/be/src/olap/lru_cache.cpp:194
15:21:03   18# doris::ShardedLRUCache::set_capacity(unsigned long) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
15:21:03   19# doris::LRUCachePolicy::adjust_capacity_weighted(double) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
15:21:03   20# doris::CacheManager::for_each_cache_refresh_capacity(double, doris::RuntimeProfile*) at /root/doris/be/src/runtime/memory/cache_manager.cpp:76
15:21:03   21# doris::Daemon::cache_adjust_capacity_thread() in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
15:21:03   22# doris::Thread::supervise_thread(void*) at /root/doris/be/src/util/thread.cpp:499
15:21:03   23# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478
15:21:03   24# __clone at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

@doris-robot
Copy link

doris-robot commented Sep 9, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Sep 9, 2024

run buildall

@shuke987
Copy link
Collaborator

shuke987 commented Sep 9, 2024

skip buildall

shuke987
shuke987 approved these changes Sep 9, 2024
View reviewed changes
Copy link
Collaborator

@shuke987 shuke987 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Sep 9, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 9, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 9, 2024

PR approved by anyone and no changes requested.

@hello-stephen hello-stephen merged commit 6a19a37 into apache:master Sep 9, 2024
@doris-robot
Copy link

doris-robot commented Sep 9, 2024

TeamCity be ut coverage result:
Function Coverage: 36.84% (9385/25478)
Line Coverage: 28.23% (77404/274169)
Region Coverage: 27.63% (39962/144647)
Branch Coverage: 24.26% (20333/83814)
Coverage Report: http://coverage.selectdb-in.cc/coverage/11ddd6afc9fb501fa086eb4ee900d145c7f81f3b_11ddd6afc9fb501fa086eb4ee900d145c7f81f3b/report/index.html

@doris-robot
Copy link

doris-robot commented Sep 9, 2024

TPC-H: Total hot run time: 37790 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 11ddd6afc9fb501fa086eb4ee900d145c7f81f3b, data reload: false

------ Round 1 ----------------------------------
q1	17617	4443	4293	4293
q2	2023	194	186	186
q3	11695	958	1152	958
q4	10506	747	699	699
q5	7766	2798	2809	2798
q6	229	139	137	137
q7	938	610	589	589
q8	9356	2081	2064	2064
q9	6968	6543	6584	6543
q10	7008	2197	2187	2187
q11	515	249	247	247
q12	396	227	219	219
q13	17758	3124	3064	3064
q14	270	248	245	245
q15	555	495	490	490
q16	542	452	427	427
q17	988	674	693	674
q18	7309	6796	6864	6796
q19	1385	1075	1001	1001
q20	687	341	337	337
q21	3967	3030	2802	2802
q22	1111	1035	1034	1034
Total cold run time: 109589 ms
Total hot run time: 37790 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4457	4341	4287	4287
q2	376	285	272	272
q3	2866	2617	2654	2617
q4	1964	1681	1719	1681
q5	5560	5709	5736	5709
q6	235	144	138	138
q7	2276	1875	1842	1842
q8	3360	3408	3450	3408
q9	8856	8834	8802	8802
q10	3551	3409	3355	3355
q11	609	525	512	512
q12	837	675	664	664
q13	15234	3267	3266	3266
q14	326	289	290	289
q15	555	486	487	486
q16	554	509	508	508
q17	1846	1534	1533	1533
q18	8188	7810	7992	7810
q19	1744	1618	1577	1577
q20	2154	1897	1937	1897
q21	5753	5646	5516	5516
q22	1118	1033	1023	1023
Total cold run time: 72419 ms
Total hot run time: 57192 ms

@doris-robot
Copy link

doris-robot commented Sep 9, 2024

TPC-DS: Total hot run time: 192218 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 11ddd6afc9fb501fa086eb4ee900d145c7f81f3b, data reload: false

query1	1266	881	861	861
query2	6442	1901	1923	1901
query3	10597	3909	3924	3909
query4	59494	24370	22990	22990
query5	5391	511	521	511
query6	415	158	170	158
query7	5771	303	296	296
query8	325	224	230	224
query9	8749	2514	2525	2514
query10	504	282	272	272
query11	17813	14980	15274	14980
query12	156	107	108	107
query13	1514	391	366	366
query14	11293	7226	7883	7226
query15	249	175	189	175
query16	7560	493	479	479
query17	1123	561	553	553
query18	2062	286	300	286
query19	292	151	145	145
query20	119	110	110	110
query21	209	102	107	102
query22	4606	4508	4686	4508
query23	34420	33485	34011	33485
query24	6006	2949	2875	2875
query25	506	379	390	379
query26	671	148	151	148
query27	1771	271	281	271
query28	3699	2030	2013	2013
query29	629	395	403	395
query30	234	148	144	144
query31	952	772	750	750
query32	85	51	51	51
query33	442	287	279	279
query34	863	476	493	476
query35	826	740	701	701
query36	1052	897	926	897
query37	143	81	85	81
query38	3919	3864	3915	3864
query39	1429	1400	1409	1400
query40	191	113	114	113
query41	46	45	45	45
query42	111	96	97	96
query43	520	480	468	468
query44	1101	749	733	733
query45	194	164	161	161
query46	1076	735	742	735
query47	1855	1783	1791	1783
query48	393	297	292	292
query49	790	493	437	437
query50	827	408	406	406
query51	7012	7115	6755	6755
query52	102	87	89	87
query53	248	173	178	173
query54	570	456	447	447
query55	75	74	75	74
query56	290	253	245	245
query57	1220	1076	1076	1076
query58	229	242	236	236
query59	2983	2824	2779	2779
query60	299	271	268	268
query61	110	104	99	99
query62	762	636	644	636
query63	221	186	182	182
query64	2809	698	647	647
query65	3233	3169	3159	3159
query66	638	329	333	329
query67	15488	15310	15214	15214
query68	2903	595	575	575
query69	400	290	275	275
query70	1208	1049	1050	1049
query71	348	279	282	279
query72	5850	4082	4030	4030
query73	744	325	327	325
query74	9159	8976	9049	8976
query75	3358	2685	2682	2682
query76	1415	1002	997	997
query77	573	328	332	328
query78	9723	8982	9110	8982
query79	1026	531	531	531
query80	748	517	512	512
query81	460	227	227	227
query82	238	132	138	132
query83	172	151	148	148
query84	256	77	74	74
query85	681	288	284	284
query86	304	281	295	281
query87	4319	4419	4199	4199
query88	2976	2278	2288	2278
query89	393	283	286	283
query90	2123	192	190	190
query91	123	119	103	103
query92	57	49	47	47
query93	1070	533	533	533
query94	733	296	298	296
query95	343	259	258	258
query96	591	264	262	262
query97	3168	3130	3055	3055
query98	224	208	259	208
query99	1517	1262	1268	1262
Total cold run time: 304193 ms
Total hot run time: 192218 ms

@doris-robot
Copy link

doris-robot commented Sep 9, 2024

ClickBench: Total hot run time: 31.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 11ddd6afc9fb501fa086eb4ee900d145c7f81f3b, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.07
query5	0.50	0.49	0.50
query6	1.13	0.73	0.74
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.54	0.50	0.48
query10	0.54	0.56	0.54
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.62	0.59	0.59
query14	1.41	1.40	1.43
query15	0.82	0.81	0.82
query16	0.37	0.38	0.37
query17	1.00	0.99	1.00
query18	0.21	0.21	0.20
query19	1.95	1.77	1.87
query20	0.01	0.01	0.01
query21	15.38	0.67	0.66
query22	4.15	6.77	2.37
query23	18.34	1.40	1.29
query24	2.15	0.23	0.21
query25	0.15	0.08	0.08
query26	0.26	0.18	0.17
query27	0.08	0.07	0.08
query28	13.24	1.01	0.99
query29	12.65	3.36	3.30
query30	0.24	0.05	0.04
query31	2.89	0.40	0.38
query32	3.24	0.47	0.48
query33	3.00	2.97	2.98
query34	16.97	4.36	4.41
query35	4.43	4.39	4.43
query36	0.65	0.48	0.48
query37	0.20	0.15	0.16
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.17	0.12	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.07 s
Total hot run time: 31.67 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hello-stephen hello-stephen hello-stephen approved these changes

+1 more reviewer

@shuke987 shuke987 shuke987 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@xinyiZzz @doris-robot @shuke987 @hello-stephen