[WIP] Try to improve spark row written performance by optimizing memory access#24
Closed
taiyang-li wants to merge 1 commit intospark_row_info_with_map_arrayfrom
Closed
[WIP] Try to improve spark row written performance by optimizing memory access#24taiyang-li wants to merge 1 commit intospark_row_info_with_map_arrayfrom
taiyang-li wants to merge 1 commit intospark_row_info_with_map_arrayfrom
Conversation
Collaborator
Author
|
I met a quite strange issue. This is disassembly code: |
taiyang-li
changed the title
lgbo-ustc
pushed a commit
that referenced
this pull request
Apr 16, 2024
Fixes integration test_reload_certificate/test.py::test_first_than_second_cert --- E Exception: Sanitizer assert found for instance ================== E WARNING: ThreadSanitizer: data race (pid=1) E Write of size 8 at 0x7b2800025d30 by thread T2 (mutexes: write M0, write M1): E #0 free <null> (clickhouse+0x709a3e5) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #1 CRYPTO_free build_docker/./contrib/openssl/crypto/mem.c:282:5 (clickhouse+0x2015f8ea) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #2 EVP_PKEY_free build_docker/./contrib/openssl/crypto/evp/p_lib.c:1809:5 (clickhouse+0x2012a751) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #3 Poco::Crypto::EVPPKey::~EVPPKey() build_docker/./base/poco/Crypto/src/EVPPKey.cpp:121:17 (clickhouse+0x1d00ffa9) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #4 DB::CertificateReloader::Data::~Data() build_docker/./src/Server/CertificateReloader.h:71:12 (clickhouse+0x194fb42d) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #5 std::__1::default_delete<DB::CertificateReloader::Data const>::operator()[abi:v15000](DB::CertificateReloader::Data const*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 (clickhouse+0x194fb42d) E #6 std::__1::__shared_ptr_pointer<DB::CertificateReloader::Data const*, std::__1::default_delete<DB::CertificateReloader::Data const>, std::__1::allocator<DB::CertificateReloader::Data const>>::__on_zero_shared() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:263:5 (clickhouse+0x194fb42d) E #7 std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 (clickhouse+0x194fade0) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #8 std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 (clickhouse+0x194fade0) E #9 std::__1::shared_ptr<DB::CertificateReloader::Data const>::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 (clickhouse+0x194fade0) E #10 std::__1::shared_ptr<DB::CertificateReloader::Data const>::operator=[abi:v15000](std::__1::shared_ptr<DB::CertificateReloader::Data const>&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:723:9 (clickhouse+0x194fade0) E #11 MultiVersion<DB::CertificateReloader::Data>::set(std::__1::unique_ptr<DB::CertificateReloader::Data const, std::__1::default_delete<DB::CertificateReloader::Data const>>&&) build_docker/./src/Common/MultiVersion.h:76:25 (clickhouse+0x194fade0) E #12 DB::CertificateReloader::tryLoad(Poco::Util::AbstractConfiguration const&) build_docker/./src/Server/CertificateReloader.cpp:83:18 (clickhouse+0x194f94ca) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #13 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6::operator()(Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool) const build_docker/./programs/server/Server.cpp:1546:45 (clickhouse+0xf384df7) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #14 decltype(std::declval<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&>()(std::declval<Poco::AutoPtr<Poco::Util::AbstractConfiguration>>(), std::declval<bool>())) std::__1::__invoke[abi:v15000]<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf3827a9) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #15 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf3827a9) E #16 std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6, void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()[abi:v15000](Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf3827a9) E #17 void std::__1::__function::__policy_invoker<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::__call_impl<std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6, void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>>(std::__1::__function::__policy_storage const*, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf3827a9) E #18 std::__1::__function::__policy_func<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()[abi:v15000](Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x19fd2cbe) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #19 std::__1::function<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()(Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x19fd2cbe) E #20 DB::ConfigReloader::reloadIfNewer(bool, bool, bool, bool) build_docker/./src/Common/Config/ConfigReloader.cpp:150:13 (clickhouse+0x19fd2cbe) E #21 DB::ConfigReloader::reload() build_docker/./src/Common/Config/ConfigReloader.h:51:21 (clickhouse+0xf38767c) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #22 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13::operator()() const build_docker/./programs/server/Server.cpp:1731:31 (clickhouse+0xf38767c) E #23 decltype(std::declval<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>()()) std::__1::__invoke[abi:v15000]<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf38767c) E #24 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf38767c) E #25 std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13, void ()>::operator()[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf38767c) E #26 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13, void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf38767c) E #27 std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x16907aa0) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #28 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x16907aa0) E #29 DB::Context::reloadConfig() const build_docker/./src/Interpreters/Context.cpp:4357:5 (clickhouse+0x16907aa0) E ClickHouse#30 DB::InterpreterSystemQuery::execute() build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:577:29 (clickhouse+0x17e78c19) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E ClickHouse#31 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1195:40 (clickhouse+0x17e3e462) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E ClickHouse#32 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1374:26 (clickhouse+0x17e39837) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E ClickHouse#33 DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:518:54 (clickhouse+0x195cc651) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E ClickHouse#34 DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2329:9 (clickhouse+0x195e8707) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E ClickHouse#35 Poco::Net::TCPServerConnection::start() build_docker/./base/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x1d00d942) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E ClickHouse#36 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115:20 (clickhouse+0x1d00e1b1) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E ClickHouse#37 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:188:14 (clickhouse+0x1d20f2e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E ClickHouse#38 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1d20d5af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E ClickHouse#39 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1d20ba69) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E E Previous atomic write of size 4 at 0x7b2800025d30 by thread T3 (mutexes: write M2): E #0 CRYPTO_DOWN_REF build_docker/./contrib/openssl/include/internal/refcount.h:51:12 (clickhouse+0x2012a6e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #1 EVP_PKEY_free build_docker/./contrib/openssl/crypto/evp/p_lib.c:1795:5 (clickhouse+0x2012a6e6) E #2 ssl_cert_clear_certs build_docker/./contrib/openssl/ssl/ssl_cert.c:246:9 (clickhouse+0x1ffafd37) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #3 ssl_cert_free build_docker/./contrib/openssl/ssl/ssl_cert.c:277:5 (clickhouse+0x1ffafd37) E #4 ossl_ssl_connection_free build_docker/./contrib/openssl/ssl/ssl_lib.c:1458:5 (clickhouse+0x1ffba6af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #5 SSL_free build_docker/./contrib/openssl/ssl/ssl_lib.c:1417:9 (clickhouse+0x1ffb920e) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #6 Poco::Net::SecureSocketImpl::reset() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:583:3 (clickhouse+0x1cfaac60) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #7 Poco::Net::SecureSocketImpl::~SecureSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:80:3 (clickhouse+0x1cfaac60) E #8 Poco::Net::SecureStreamSocketImpl::~SecureStreamSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:52:1 (clickhouse+0x1cfb15dd) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #9 Poco::Net::SecureStreamSocketImpl::~SecureStreamSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:43:1 (clickhouse+0x1cfb15dd) E #10 Poco::RefCountedObject::release() const build_docker/./base/poco/Foundation/include/Poco/RefCountedObject.h:86:13 (clickhouse+0x1cffc81e) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #11 Poco::Net::Socket::~Socket() build_docker/./base/poco/Net/src/Socket.cpp:68:10 (clickhouse+0x1cffc81e) E #12 Poco::Net::StreamSocket::~StreamSocket() build_docker/./base/poco/Net/src/StreamSocket.cpp:63:1 (clickhouse+0x1d009c39) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #13 Poco::Net::TCPConnectionNotification::~TCPConnectionNotification() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:43:2 (clickhouse+0x1d00ef50) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #14 Poco::Net::TCPConnectionNotification::~TCPConnectionNotification() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:42:2 (clickhouse+0x1d00ef50) E #15 Poco::RefCountedObject::release() const build_docker/./base/poco/Foundation/include/Poco/RefCountedObject.h:86:13 (clickhouse+0x1d00e203) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #16 Poco::AutoPtr<Poco::Notification>::~AutoPtr() build_docker/./base/poco/Foundation/include/Poco/AutoPtr.h:91:19 (clickhouse+0x1d00e203) E #17 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:122:3 (clickhouse+0x1d00e203) E #18 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:188:14 (clickhouse+0x1d20f2e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #19 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1d20d5af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #20 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1d20ba69) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E
taiyang-li
pushed a commit
that referenced
this pull request
Jul 21, 2025
…ary bridge cleanup
<details>
<summary>stacktrace</summary>
#5 DB::ContextSharedMutex::lockSharedImpl (this=this@entry=0x7ffff5b96040) at /src/ch/clickhouse/src/Interpreters/Context.cpp:1036
watch = {start_ns = 98678735300141, stop_ns = 0, clock_type = 4, is_running = true}
increment = {what = <optimized out>, amount = 1}
#6 0x00005555699352d6 in DB::SharedMutexHelper<DB::ContextSharedMutex, DB::SharedMutex>::lock_shared (this=0x7ffff5b96040) at /src/ch/clickhouse/src/Common/SharedMutexHelper.h:70
#7 DB::SharedLockGuard<DB::ContextSharedMutex>::SharedLockGuard (mutex_=..., this=<optimized out>) at /src/ch/clickhouse/src/Common/SharedLockGuard.h:19
#8 DB::ContextSharedPart::getConfigRef (this=0x7ffff5b96000) at /src/ch/clickhouse/src/Interpreters/Context.cpp:760
#9 DB::Context::getConfigRef (this=<optimized out>) at /src/ch/clickhouse/src/Interpreters/Context.cpp:1590
#10 0x000055556450941c in DB::ProxyConfigurationResolverProvider::get (request_protocol=DB::ProxyConfiguration::Protocol::HTTP) at /src/ch/clickhouse/src/Common/ProxyConfigurationResolverProvider.cpp:161
#11 0x00005555669ea75f in DB::BuilderRWBufferFromHTTP::create (this=0x7fffffffa848, credentials_=...) at /src/ch/clickhouse/src/IO/ReadWriteBufferFromHTTP.cpp:814
#12 0x000055556cea0035 in DB::ExternalDictionaryLibraryBridgeHelper::bridgeHandShake (this=0x7ffd60c4ca18) at /src/ch/clickhouse/src/BridgeHelper/ExternalDictionaryLibraryBridgeHelper.cpp:79
#13 0x000055556cea1f34 in DB::ExternalDictionaryLibraryBridgeHelper::removeLibrary (this=0x7ffff5b96044) at /src/ch/clickhouse/src/BridgeHelper/ExternalDictionaryLibraryBridgeHelper.cpp:186
#14 0x0000555566cb0248 in DB::LibraryDictionarySource::~LibraryDictionarySource (this=0x7ffd60c78318) at /src/ch/clickhouse/src/Dictionaries/LibraryDictionarySource.cpp:70
#15 0x000055556708d775 in std::__1::__shared_count::__release_shared[abi:ne190107]() (this=0x7ffd60c78300) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:156
#16 std::__1::__shared_weak_count::__release_shared[abi:ne190107]() (this=0x7ffd60c78300) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:185
#17 std::__1::shared_ptr<DB::IDictionarySource>::~shared_ptr[abi:ne190107]() (this=0x7ffd60c605b0) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:668
#18 DB::HashedDictionary<(DB::DictionaryKeyType)0, false, false>::~HashedDictionary (this=0x7ffd60c60398) at /src/ch/clickhouse/src/Dictionaries/HashedDictionary.h:378
#19 0x0000555569a9c477 in std::__1::__shared_count::__release_shared[abi:ne190107]() (this=0x7ffd60c60380) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:156
#20 std::__1::__shared_weak_count::__release_shared[abi:ne190107]() (this=0x7ffd60c60380) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:185
#21 std::__1::shared_ptr<DB::IExternalLoadable>::~shared_ptr[abi:ne190107]() (this=0x7ffff4832560) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:668
#22 DB::ExternalLoader::LoadingDispatcher::Info::~Info (this=0x7ffff4832548) at /src/ch/clickhouse/src/Interpreters/ExternalLoader.cpp:719
#23 std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, DB::ExternalLoader::LoadingDispatcher::Info>::~pair (this=0x7ffff4832530, this@entry=0x7ffd60c60390) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__utility/pair.h:65
#24 __destroy_at<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, DB::ExternalLoader::LoadingDispatcher::Info>, 0> (__loc=__loc@entry=0x7ffff4832530) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__memory/construct_at.h:67
#25 0x0000555569a9c628 in destroy<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, DB::ExternalLoader::LoadingDispatcher::Info>, void, 0> (__p=0x7ffff5b96044, __p@entry=0x7ffff4832530) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__memory/allocator_traits.h:339
#28 std::__1::unique_ptr<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DB::ExternalLoader::LoadingDispatcher::Info>, void*>, std::__1::__hash_node_destructor<std::__1::allocator<std::__1::__hash_node<std::__1::__hash_value_type<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, DB::ExternalLoader::LoadingDispatcher::Info>, void*> > > >::~unique_ptr[abi:ne190107]() (this=<optimized out>) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:261
ClickHouse#31 DB::ExternalLoader::LoadingDispatcher::setConfiguration (this=this@entry=0x7ffecad4c300, new_configs=...) at /src/ch/clickhouse/src/Interpreters/ExternalLoader.cpp:495
ClickHouse#32 0x0000555569a9116d in DB::ExternalLoader::reloadConfig (this=0x7ffecb3fd740, repository_name=...) at /src/ch/clickhouse/src/Interpreters/ExternalLoader.cpp:1533
ClickHouse#33 DB::ExternalLoader::addConfigRepository(std::__1::unique_ptr<DB::IExternalLoaderConfigRepository, std::__1::default_delete<DB::IExternalLoaderConfigRepository> >) const::$_0::operator()() const (this=<optimized out>) at /src/ch/clickhouse/src/Interpreters/ExternalLoader.cpp:1334
ClickHouse#34 std::__1::__invoke[abi:ne190107]<DB::ExternalLoader::addConfigRepository(std::__1::unique_ptr<DB::IExternalLoaderConfigRepository, std::__1::default_delete<DB::IExternalLoaderConfigRepository> >) const::$_0&>(DB::ExternalLoader::addConfigRepository(std::__1::unique_ptr<DB::IExternalLoaderConfigRepository, std::__1::default_delete<DB::IExternalLoaderConfigRepository> >) const::$_0&) (__f=...) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__type_traits/invoke.h:149
ClickHouse#35 std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:ne190107]<DB::ExternalLoader::addConfigRepository(std::__1::unique_ptr<DB::IExternalLoaderConfigRepository, std::__1::default_delete<DB::IExternalLoaderConfigRepository> >) const::$_0&>(DB::ExternalLoader::addConfigRepository(std::__1::unique_ptr<DB::IExternalLoaderConfigRepository, std::__1::default_delete<DB::IExternalLoaderConfigRepository> >) const::$_0&) (__args=...) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__type_traits/invoke.h:224
ClickHouse#36 std::__1::__function::__default_alloc_func<DB::ExternalLoader::addConfigRepository(std::__1::unique_ptr<DB::IExternalLoaderConfigRepository, std::__1::default_delete<DB::IExternalLoaderConfigRepository> >) const::$_0, void ()>::operator()[abi:ne190107]() (this=<optimized out>) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__functional/function.h:210
ClickHouse#37 std::__1::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__1::__function::__default_alloc_func<DB::ExternalLoader::addConfigRepository(std::__1::unique_ptr<DB::IExternalLoaderConfigRepository, std::__1::default_delete<DB::IExternalLoaderConfigRepository> >) const::$_0, void ()> ClickHouse#38 0x0000555569966aaa in std::__1::__function::__policy_func<void ()>::operator()[abi:ne190107]() const (this=0x7ffff5b96044) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__functional/function.h:716
ClickHouse#39 std::__1::function<void()>::operator() (this=0x7ffff5b96044) at /src/ch/clickhouse/contrib/llvm-project/libcxx/include/__functional/function.h:989
ClickHouse#40 BasicScopeGuard<std::__1::function<void()> >::invoke (this=0x7ffff5b96044) at /src/ch/clickhouse/base/base/../base/scope_guard.h:101
ClickHouse#41 BasicScopeGuard<std::__1::function<void()> >::reset (this=0x7ffff5b96044) at /src/ch/clickhouse/base/base/../base/scope_guard.h:63
ClickHouse#42 DB::ContextSharedPart::shutdown (this=0x7ffff5b96000) at /src/ch/clickhouse/src/Interpreters/Context.cpp:909
ClickHouse#43 0x00005555644cf5cb in DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_3::operator()() const (this=this@entry=0x7fffffffc870) at /src/ch/clickhouse/programs/server/Server.cpp:1257
ClickHouse#44 0x00005555644b4439 in BasicScopeGuard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_3>::invoke (this=0x7fffffffc870) at /src/ch/clickhouse/base/base/../base/scope_guard.h:101
ClickHouse#45 BasicScopeGuard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_3>::~BasicScopeGuard (this=0x7fffffffc870) at /src/ch/clickhouse/base/base/../base/scope_guard.h:50
ClickHouse#46 DB::Server::main (this=0x7fffffffe4c8) at /src/ch/clickhouse/programs/server/Server.cpp:2788
ClickHouse#47 0x0000555572cbbee9 in Poco::Util::Application::run (this=0x7fffffffe4c8) at /src/ch/clickhouse/base/poco/Util/src/Application.cpp:315
ClickHouse#48 0x000055556449c390 in DB::Server::run (this=0x7fffffffe4c8) at /src/ch/clickhouse/programs/server/Server.cpp:610
ClickHouse#49 0x0000555564499aec in mainEntryClickHouseServer (argc=6, argv=0x7ffff67ecdc0) at /src/ch/clickhouse/programs/server/Server.cpp:397
ClickHouse#50 0x000055555dc994c2 in main (argc_=<optimized out>, argv_=<optimized out>) at /src/ch/clickhouse/programs/main.cpp:340
</details>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Write entire row to SparkRowInfo one by one instead os write entire column to SparkRowInfo.
This is just a POC, and it is fully completed yet.
The benchmark shows that before optimization it takes 240ms, after optimization it takes 170ms.