disconnect broken-connection by CheSema · Pull Request #83056 · ClickHouse/ClickHouse

CheSema · 2025-07-02T09:08:37Z

ConnectionPoolWithFailover doing bad thing. It holds connection in whatever state. Broken connections are being put back to the pool at the end and they are stored there as well.

Related to:
It is not allowed to use read buffer after it thrown an exceptoin.
#81951

The error looks like this

[ 4402 ] {BgDistSchPool::279e6051-c0b8-43d3-b2f9-c4805f7fef25} <Fatal> : Logical error: \'ReadBuffer is canceled. Can\'t read from it.\'.
 [ 4402 ] {BgDistSchPool::279e6051-c0b8-43d3-b2f9-c4805f7fef25} <Fatal> : Stack trace (when copying this message, always include the lines below):
 [ 4398 ] {BgDistSchPool::55106476-f3e6-473c-ab8a-aba052c08be9} <Fatal> : Logical error: \'ReadBuffer is canceled. Can\'t read from it.\'.
 [ 4398 ] {BgDistSchPool::55106476-f3e6-473c-ab8a-aba052c08be9} <Fatal> : Stack trace (when copying this message, always include the lines below):
 [ 5365 ] {} <Fatal> BaseDaemon: ########## Short fault info ############
 [ 5365 ] {} <Fatal> BaseDaemon: (version 25.7.1.1, build id: 4E04B638C5458A4840A1F52431ACE2DDE863AE41, git hash: fcad1e84875dd2ca2b749c71885c321a2ddec427, architecture: x86_64) (from thread 4402) Received signal 6
 [ 5365 ] {} <Fatal> BaseDaemon: Signal description: Aborted
 [ 5365 ] {} <Fatal> BaseDaemon: 
 [ 5365 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007feb4e1059fd 0x00007feb4e0b1476 0x00007feb4e0977f3 0x000055a8bbe38267 0x000055a8c593bc3c 0x000055a8c593c097 0x000055a8bc014d1f 0x000055a8d18f6cd7 0x000055a8d1915247 0x000055a8d1925b1d 0x000055a8d19269e9 0x000055a8d1927cff 0x000055a8d192514a 0x000055a8d19257b0 0x000055a8d0d55053 0x000055a8d0d4f0a2 0x000055a8d0d4f740 0x000055a8d0d58ee2 0x000055a8cc3075e9 0x000055a8cc30c80f 0x000055a8cc30d4ce 0x000055a8c5ae96c3 0x000055a8c5af241c 0x000055a8bbe31428 0x00007feb4e103ac3 0x00007feb4e195850
 [ 5365 ] {} <Fatal> BaseDaemon: ########################################
 [ 5366 ] {} <Fatal> BaseDaemon: ########## Short fault info ############
 [ 5366 ] {} <Fatal> BaseDaemon: (version 25.7.1.1, build id: 4E04B638C5458A4840A1F52431ACE2DDE863AE41, git hash: fcad1e84875dd2ca2b749c71885c321a2ddec427, architecture: x86_64) (from thread 4398) Received signal 6
 [ 5366 ] {} <Fatal> BaseDaemon: Signal description: Aborted
 [ 5366 ] {} <Fatal> BaseDaemon: 
 [ 5366 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007feb4e1059fd 0x00007feb4e0b1476 0x00007feb4e0977f3 0x000055a8bbe38267 0x000055a8c593bc3c 0x000055a8c593c097 0x000055a8bc014d1f 0x000055a8d18f6cd7 0x000055a8d1915247 0x000055a8d1925b1d 0x000055a8d19269e9 0x000055a8d1927cff 0x000055a8d192514a 0x000055a8d19257b0 0x000055a8d0d55053 0x000055a8d0d4f0a2 0x000055a8d0d4f740 0x000055a8d0d58ee2 0x000055a8cc3075e9 0x000055a8cc30c80f 0x000055a8cc30d4ce 0x000055a8c5ae96c3 0x000055a8c5af241c 0x000055a8bbe31428 0x00007feb4e103ac3 0x00007feb4e195850
 [ 5366 ] {} <Fatal> BaseDaemon: ########################################
 [ 5365 ] {} <Fatal> BaseDaemon: (version 25.7.1.1, build id: 4E04B638C5458A4840A1F52431ACE2DDE863AE41, git hash: fcad1e84875dd2ca2b749c71885c321a2ddec427) (from thread 4402) (query_id: BgDistSchPool::279e6051-c0b8-43d3-b2f9-c4805f7fef25) (query: ) Received signal Aborted (6)
 [ 5366 ] {} <Fatal> BaseDaemon: (version 25.7.1.1, build id: 4E04B638C5458A4840A1F52431ACE2DDE863AE41, git hash: fcad1e84875dd2ca2b749c71885c321a2ddec427) (from thread 4398) (query_id: BgDistSchPool::55106476-f3e6-473c-ab8a-aba052c08be9) (query: ) Received signal Aborted (6)
 [ 5365 ] {} <Fatal> BaseDaemon: 
 [ 5366 ] {} <Fatal> BaseDaemon: 
 [ 5365 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007feb4e1059fd 0x00007feb4e0b1476 0x00007feb4e0977f3 0x000055a8bbe38267 0x000055a8c593bc3c 0x000055a8c593c097 0x000055a8bc014d1f 0x000055a8d18f6cd7 0x000055a8d1915247 0x000055a8d1925b1d 0x000055a8d19269e9 0x000055a8d1927cff 0x000055a8d192514a 0x000055a8d19257b0 0x000055a8d0d55053 0x000055a8d0d4f0a2 0x000055a8d0d4f740 0x000055a8d0d58ee2 0x000055a8cc3075e9 0x000055a8cc30c80f 0x000055a8cc30d4ce 0x000055a8c5ae96c3 0x000055a8c5af241c 0x000055a8bbe31428 0x00007feb4e103ac3 0x00007feb4e195850
 [ 5366 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007feb4e1059fd 0x00007feb4e0b1476 0x00007feb4e0977f3 0x000055a8bbe38267 0x000055a8c593bc3c 0x000055a8c593c097 0x000055a8bc014d1f 0x000055a8d18f6cd7 0x000055a8d1915247 0x000055a8d1925b1d 0x000055a8d19269e9 0x000055a8d1927cff 0x000055a8d192514a 0x000055a8d19257b0 0x000055a8d0d55053 0x000055a8d0d4f0a2 0x000055a8d0d4f740 0x000055a8d0d58ee2 0x000055a8cc3075e9 0x000055a8cc30c80f 0x000055a8cc30d4ce 0x000055a8c5ae96c3 0x000055a8c5af241c 0x000055a8bbe31428 0x00007feb4e103ac3 0x00007feb4e195850
 [ 5365 ] {} <Fatal> BaseDaemon: 5. ? @ 0x00000000000969fd
 [ 5365 ] {} <Fatal> BaseDaemon: 6. ? @ 0x0000000000042476
 [ 5365 ] {} <Fatal> BaseDaemon: 7. ? @ 0x00000000000287f3
 [ 5366 ] {} <Fatal> BaseDaemon: 5. ? @ 0x00000000000969fd
 [ 5366 ] {} <Fatal> BaseDaemon: 6. ? @ 0x0000000000042476
 [ 5366 ] {} <Fatal> BaseDaemon: 7. ? @ 0x00000000000287f3
 [ 5365 ] {} <Fatal> BaseDaemon: 8. ___interceptor_abort @ 0x0000000008e03267
 [ 5366 ] {} <Fatal> BaseDaemon: 8. ___interceptor_abort @ 0x0000000008e03267
 [ 5366 ] {} <Fatal> BaseDaemon: 9. ./ci/tmp/build/./src/Common/Exception.cpp:50: DB::abortOnFailedAssertion(String const&, void* const*, unsigned long, unsigned long) @ 0x0000000012906c3c
 [ 5365 ] {} <Fatal> BaseDaemon: 9. ./ci/tmp/build/./src/Common/Exception.cpp:50: DB::abortOnFailedAssertion(String const&, void* const*, unsigned long, unsigned long) @ 0x0000000012906c3c
 [ 5366 ] {} <Fatal> BaseDaemon: 10. ./ci/tmp/build/./src/Common/Exception.cpp:56: DB::abortOnFailedAssertion(String const&) @ 0x0000000012907097
 [ 5365 ] {} <Fatal> BaseDaemon: 10. ./ci/tmp/build/./src/Common/Exception.cpp:56: DB::abortOnFailedAssertion(String const&) @ 0x0000000012907097
 [ 5366 ] {} <Fatal> BaseDaemon: 11. DB::ReadBuffer::next() @ 0x0000000008fdfd1f
 [ 5365 ] {} <Fatal> BaseDaemon: 11. DB::ReadBuffer::next() @ 0x0000000008fdfd1f
 [ 5366 ] {} <Fatal> BaseDaemon: 12.0. inlined from ./src/IO/ReadBuffer.h:114: DB::ReadBuffer::eof()
 [ 5366 ] {} <Fatal> BaseDaemon: 12.1. inlined from ./src/IO/VarInt.h:77: void DB::varint_impl::readVarUInt<true>(unsigned long&, DB::ReadBuffer&)
 [ 5366 ] {} <Fatal> BaseDaemon: 12.2. inlined from ./src/IO/VarInt.h:96: DB::readVarUInt(unsigned long&, DB::ReadBuffer&)
 [ 5366 ] {} <Fatal> BaseDaemon: 12. ./ci/tmp/build/./src/Client/Connection.cpp:783: DB::Connection::getTablesStatus(DB::ConnectionTimeouts const&, DB::TablesStatusRequest const&) @ 0x000000001e8c1cd7
 [ 5365 ] {} <Fatal> BaseDaemon: 12.0. inlined from ./src/IO/ReadBuffer.h:114: DB::ReadBuffer::eof()
 [ 5365 ] {} <Fatal> BaseDaemon: 12.1. inlined from ./src/IO/VarInt.h:77: void DB::varint_impl::readVarUInt<true>(unsigned long&, DB::ReadBuffer&)
 [ 5365 ] {} <Fatal> BaseDaemon: 12.2. inlined from ./src/IO/VarInt.h:96: DB::readVarUInt(unsigned long&, DB::ReadBuffer&)
 [ 5365 ] {} <Fatal> BaseDaemon: 12. ./ci/tmp/build/./src/Client/Connection.cpp:783: DB::Connection::getTablesStatus(DB::ConnectionTimeouts const&, DB::TablesStatusRequest const&) @ 0x000000001e8c1cd7
 [ 5365 ] {} <Fatal> BaseDaemon: 13. ./ci/tmp/build/./src/Client/ConnectionEstablisher.cpp:74: DB::ConnectionEstablisher::run(PoolWithFailoverBase<DB::IConnectionPool>::TryResult&, String&, bool) @ 0x000000001e8e0247
 [ 5366 ] {} <Fatal> BaseDaemon: 13. ./ci/tmp/build/./src/Client/ConnectionEstablisher.cpp:74: DB::ConnectionEstablisher::run(PoolWithFailoverBase<DB::IConnectionPool>::TryResult&, String&, bool) @ 0x000000001e8e0247
 [ 5365 ] {} <Fatal> BaseDaemon: 14. ./ci/tmp/build/./src/Client/ConnectionPoolWithFailover.cpp:276: DB::ConnectionPoolWithFailover::tryGetEntry(std::shared_ptr<DB::IConnectionPool> const&, DB::ConnectionTimeouts const&, String&, DB::Settings const&, DB::QualifiedTableName const*, std::function<void (int, Poco::Timespan, DB::AsyncEventTimeoutType, String const&, unsigned int)>, bool) @ 0x000000001e8f0b1d
 [ 5366 ] {} <Fatal> BaseDaemon: 14. ./ci/tmp/build/./src/Client/ConnectionPoolWithFailover.cpp:276: DB::ConnectionPoolWithFailover::tryGetEntry(std::shared_ptr<DB::IConnectionPool> const&, DB::ConnectionTimeouts const&, String&, DB::Settings const&, DB::QualifiedTableName const*, std::function<void (int, Poco::Timespan, DB::AsyncEventTimeoutType, String const&, unsigned int)>, bool) @ 0x000000001e8f0b1d
 [ 5365 ] {} <Fatal> BaseDaemon: 15.0. inlined from ./ci/tmp/build/./src/Client/ConnectionPoolWithFailover.cpp:179: operator()
 [ 5366 ] {} <Fatal> BaseDaemon: 15.0. inlined from ./ci/tmp/build/./src/Client/ConnectionPoolWithFailover.cpp:179: operator()
 [ 5365 ] {} <Fatal> BaseDaemon: 15.1. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:149: decltype(std::declval<DB::ConnectionPoolWithFailover::getManyCheckedForInsert(DB::ConnectionTimeouts const&, DB::Settings const&, DB::PoolMode, DB::QualifiedTableName const&)::$_0&>()(std::declval<std::shared_ptr<DB::IConnectionPool> const&>(), std::declval<String&>())) std::__invoke[abi:ne190107]<DB::ConnectionPoolWithFailover::getManyCheckedForInsert(DB::ConnectionTimeouts const&, DB::Settings const&, DB::PoolMode, DB::QualifiedTableName const&)::$_0&, std::shared_ptr<DB::IConnectionPool> const&, String&>(DB::ConnectionPoolWithFailover::getManyCheckedForInsert(DB::ConnectionTimeouts const&, DB::Settings const&, DB::PoolMode, DB::QualifiedTableName const&)::$_0&, std::shared_ptr<DB::IConnectionPool> const&, String&)
 [ 5366 ] {} <Fatal> BaseDaemon: 15.1. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:149: decltype(std::declval<DB::ConnectionPoolWithFailover::getManyCheckedForInsert(DB::ConnectionTimeouts const&, DB::Settings const&, DB::PoolMode, DB::QualifiedTableName const&)::$_0&>()(std::declval<std::shared_ptr<DB::IConnectionPool> const&>(), std::declval<String&>())) std::__invoke[abi:ne190107]<DB::ConnectionPoolWithFailover::getManyCheckedForInsert(DB::ConnectionTimeouts const&, DB::Settings const&, DB::PoolMode, DB::QualifiedTableName const&)::$_0&, std::shared_ptr<DB::IConnectionPool> const&, String&>(DB::ConnectionPoolWithFailover::getManyCheckedForInsert(DB::ConnectionTimeouts const&, DB::Settings const&, DB::PoolMode, DB::QualifiedTableName const&)::$_0&, std::shared_ptr<DB::IConnectionPool> const&, String&)
 [ 5365 ] {} <Fatal> BaseDaemon: 15.2. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:216: PoolWithFailoverBase<DB::IConnectionPool>::TryResult std::__invoke_void_return_wrapper<PoolWithFailoverBase<DB::IConnectionPool>::TryResult, false>::__call[abi:ne190107]<DB::ConnectionPoolWithFailover::getManyCheckedForInsert(DB::ConnectionTimeouts const&, DB::Settings const&, DB::PoolMode, DB::QualifiedTableName const&)::$_0&, std::shared_ptr<DB::IConnectionPool> const&, String&>(DB::ConnectionPoolWithFailover::getManyCheckedForInsert(DB::ConnectionTimeouts const&, DB::Settings const&, DB::PoolMode, DB::QualifiedTableName const&)::$_0&, std::shared_ptr<DB::IConnectionPool> const&, String&)
 [ 5365 ] {} <Fatal> BaseDaemon: 15.3. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:210: ?
 [ 5366 ] {} <Fatal> BaseDaemon: 15.2. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:216: PoolWithFailoverBase<DB::IConnectionPool>::TryResult std::__invoke_void_return_wrapper<PoolWithFailoverBase<DB::IConnectionPool>::TryResult, false>::__call[abi:ne190107]<DB::ConnectionPoolWithFailover::getManyCheckedForInsert(DB::ConnectionTimeouts const&, DB::Settings const&, DB::PoolMode, DB::QualifiedTableName const&)::$_0&, std::shared_ptr<DB::IConnectionPool> const&, String&>(DB::ConnectionPoolWithFailover::getManyCheckedForInsert(DB::ConnectionTimeouts const&, DB::Settings const&, DB::PoolMode, DB::QualifiedTableName const&)::$_0&, std::shared_ptr<DB::IConnectionPool> const&, String&)
 [ 5365 ] {} <Fatal> BaseDaemon: 15. ./contrib/llvm-project/libcxx/include/__functional/function.h:610: ? @ 0x000000001e8f19e9
 [ 5366 ] {} <Fatal> BaseDaemon: 15.3. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:210: ?
 [ 5366 ] {} <Fatal> BaseDaemon: 15. ./contrib/llvm-project/libcxx/include/__functional/function.h:610: ? @ 0x000000001e8f19e9
 [ 5365 ] {} <Fatal> BaseDaemon: 16.0. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ?
 [ 5365 ] {} <Fatal> BaseDaemon: 16.1. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:989: ?
 [ 5366 ] {} <Fatal> BaseDaemon: 16.0. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ?
 [ 5366 ] {} <Fatal> BaseDaemon: 16.1. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:989: ?
 [ 5366 ] {} <Fatal> BaseDaemon: 16. ./src/Common/PoolWithFailoverBase.h:310: PoolWithFailoverBase<DB::IConnectionPool>::getMany(unsigned long, unsigned long, unsigned long, unsigned long, bool, bool, std::function<PoolWithFailoverBase<DB::IConnectionPool>::TryResult (std::shared_ptr<DB::IConnectionPool> const&, String&)> const&, std::function<Priority (unsigned long)> const&) @ 0x000000001e8f2cff
 [ 5365 ] {} <Fatal> BaseDaemon: 16. ./src/Common/PoolWithFailoverBase.h:310: PoolWithFailoverBase<DB::IConnectionPool>::getMany(unsigned long, unsigned long, unsigned long, unsigned long, bool, bool, std::function<PoolWithFailoverBase<DB::IConnectionPool>::TryResult (std::shared_ptr<DB::IConnectionPool> const&, String&)> const&, std::function<Priority (unsigned long)> const&) @ 0x000000001e8f2cff
 [ 5365 ] {} <Fatal> BaseDaemon: 17. ./ci/tmp/build/./src/Client/ConnectionPoolWithFailover.cpp:237: DB::ConnectionPoolWithFailover::getManyImpl(DB::Settings const&, DB::PoolMode, std::function<PoolWithFailoverBase<DB::IConnectionPool>::TryResult (std::shared_ptr<DB::IConnectionPool> const&, String&)> const&, std::optional<bool>, std::function<Priority (unsigned long)>, bool) @ 0x000000001e8f014a
 [ 5366 ] {} <Fatal> BaseDaemon: 17. ./ci/tmp/build/./src/Client/ConnectionPoolWithFailover.cpp:237: DB::ConnectionPoolWithFailover::getManyImpl(DB::Settings const&, DB::PoolMode, std::function<PoolWithFailoverBase<DB::IConnectionPool>::TryResult (std::shared_ptr<DB::IConnectionPool> const&, String&)> const&, std::optional<bool>, std::function<Priority (unsigned long)>, bool) @ 0x000000001e8f014a
 [ 5366 ] {} <Fatal> BaseDaemon: 18. ./ci/tmp/build/./src/Client/ConnectionPoolWithFailover.cpp:181: DB::ConnectionPoolWithFailover::getManyCheckedForInsert(DB::ConnectionTimeouts const&, DB::Settings const&, DB::PoolMode, DB::QualifiedTableName const&) @ 0x000000001e8f07b0
 [ 5365 ] {} <Fatal> BaseDaemon: 18. ./ci/tmp/build/./src/Client/ConnectionPoolWithFailover.cpp:181: DB::ConnectionPoolWithFailover::getManyCheckedForInsert(DB::ConnectionTimeouts const&, DB::Settings const&, DB::PoolMode, DB::QualifiedTableName const&) @ 0x000000001e8f07b0
 [ 5366 ] {} <Fatal> BaseDaemon: 19. ./ci/tmp/build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:437: DB::DistributedAsyncInsertDirectoryQueue::processFile(String&, DB::SettingsChanges const&) @ 0x000000001dd20053
 [ 5365 ] {} <Fatal> BaseDaemon: 19. ./ci/tmp/build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:437: DB::DistributedAsyncInsertDirectoryQueue::processFile(String&, DB::SettingsChanges const&) @ 0x000000001dd20053
 [ 5365 ] {} <Fatal> BaseDaemon: 20. ./ci/tmp/build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:396: DB::DistributedAsyncInsertDirectoryQueue::processFiles(bool, DB::SettingsChanges const&) @ 0x000000001dd1a0a2
 [ 5366 ] {} <Fatal> BaseDaemon: 20. ./ci/tmp/build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:396: DB::DistributedAsyncInsertDirectoryQueue::processFiles(bool, DB::SettingsChanges const&) @ 0x000000001dd1a0a2
 [ 5365 ] {} <Fatal> BaseDaemon: 21. ./ci/tmp/build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:217: DB::DistributedAsyncInsertDirectoryQueue::run() @ 0x000000001dd1a740
 [ 5366 ] {} <Fatal> BaseDaemon: 21. ./ci/tmp/build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:217: DB::DistributedAsyncInsertDirectoryQueue::run() @ 0x000000001dd1a740
 [ 5365 ] {} <Fatal> BaseDaemon: 22.0. inlined from ./ci/tmp/build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:152: operator()
 [ 5366 ] {} <Fatal> BaseDaemon: 22.0. inlined from ./ci/tmp/build/./src/Storages/Distributed/DistributedAsyncInsertDirectoryQueue.cpp:152: operator()
 [ 5365 ] {} <Fatal> BaseDaemon: 22.1. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:149: decltype(std::declval<DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::shared_ptr<DB::IDisk> const&, String const&, std::shared_ptr<DB::ConnectionPoolWithFailover>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0&>()()) std::__invoke[abi:ne190107]<DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::shared_ptr<DB::IDisk> const&, String const&, std::shared_ptr<DB::ConnectionPoolWithFailover>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0&>(DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::shared_ptr<DB::IDisk> const&, String const&, std::shared_ptr<DB::ConnectionPoolWithFailover>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0&)
 [ 5366 ] {} <Fatal> BaseDaemon: 22.1. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:149: decltype(std::declval<DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::shared_ptr<DB::IDisk> const&, String const&, std::shared_ptr<DB::ConnectionPoolWithFailover>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0&>()()) std::__invoke[abi:ne190107]<DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::shared_ptr<DB::IDisk> const&, String const&, std::shared_ptr<DB::ConnectionPoolWithFailover>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0&>(DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::shared_ptr<DB::IDisk> const&, String const&, std::shared_ptr<DB::ConnectionPoolWithFailover>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0&)
 [ 5365 ] {} <Fatal> BaseDaemon: 22.2. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:224: void std::__invoke_void_return_wrapper<void, true>::__call[abi:ne190107]<DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::shared_ptr<DB::IDisk> const&, String const&, std::shared_ptr<DB::ConnectionPoolWithFailover>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0&>(DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::shared_ptr<DB::IDisk> const&, String const&, std::shared_ptr<DB::ConnectionPoolWithFailover>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0&)
 [ 5366 ] {} <Fatal> BaseDaemon: 22.2. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:224: void std::__invoke_void_return_wrapper<void, true>::__call[abi:ne190107]<DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::shared_ptr<DB::IDisk> const&, String const&, std::shared_ptr<DB::ConnectionPoolWithFailover>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0&>(DB::DistributedAsyncInsertDirectoryQueue::DistributedAsyncInsertDirectoryQueue(DB::StorageDistributed&, std::shared_ptr<DB::IDisk> const&, String const&, std::shared_ptr<DB::ConnectionPoolWithFailover>, DB::ActionBlocker&, DB::BackgroundSchedulePool&)::$_0&)
 [ 5365 ] {} <Fatal> BaseDaemon: 22.3. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:210: ?
 [ 5366 ] {} <Fatal> BaseDaemon: 22.3. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:210: ?
 [ 5365 ] {} <Fatal> BaseDaemon: 22. ./contrib/llvm-project/libcxx/include/__functional/function.h:610: ? @ 0x000000001dd23ee2
 [ 5366 ] {} <Fatal> BaseDaemon: 22. ./contrib/llvm-project/libcxx/include/__functional/function.h:610: ? @ 0x000000001dd23ee2
 [ 5365 ] {} <Fatal> BaseDaemon: 23.0. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ?
 [ 5366 ] {} <Fatal> BaseDaemon: 23.0. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ?
 [ 5365 ] {} <Fatal> BaseDaemon: 23.1. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:989: ?
 [ 5366 ] {} <Fatal> BaseDaemon: 23.1. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:989: ?
 [ 5365 ] {} <Fatal> BaseDaemon: 23. ./ci/tmp/build/./src/Core/BackgroundSchedulePool.cpp:138: DB::BackgroundSchedulePoolTaskInfo::execute(DB::BackgroundSchedulePool&) @ 0x00000000192d25e9
 [ 5366 ] {} <Fatal> BaseDaemon: 23. ./ci/tmp/build/./src/Core/BackgroundSchedulePool.cpp:138: DB::BackgroundSchedulePoolTaskInfo::execute(DB::BackgroundSchedulePool&) @ 0x00000000192d25e9
 [ 5365 ] {} <Fatal> BaseDaemon: 24. ./ci/tmp/build/./src/Core/BackgroundSchedulePool.cpp:356: DB::BackgroundSchedulePool::threadFunction() @ 0x00000000192d780f
 [ 5366 ] {} <Fatal> BaseDaemon: 24. ./ci/tmp/build/./src/Core/BackgroundSchedulePool.cpp:356: DB::BackgroundSchedulePool::threadFunction() @ 0x00000000192d780f
 [ 5366 ] {} <Fatal> BaseDaemon: 25.0. inlined from ./ci/tmp/build/./src/Core/BackgroundSchedulePool.cpp:219: operator()
 [ 5365 ] {} <Fatal> BaseDaemon: 25.0. inlined from ./ci/tmp/build/./src/Core/BackgroundSchedulePool.cpp:219: operator()
 [ 5366 ] {} <Fatal> BaseDaemon: 25.1. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:149: decltype(std::declval<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1&>()()) std::__invoke[abi:ne190107]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1&)
 [ 5365 ] {} <Fatal> BaseDaemon: 25.1. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:149: decltype(std::declval<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1&>()()) std::__invoke[abi:ne190107]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1&)
 [ 5366 ] {} <Fatal> BaseDaemon: 25.2. inlined from ./contrib/llvm-project/libcxx/include/tuple:1354: decltype(auto) std::__apply_tuple_impl[abi:ne190107]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1&, std::tuple<>&, std::__tuple_indices<...>)
 [ 5365 ] {} <Fatal> BaseDaemon: 25.2. inlined from ./contrib/llvm-project/libcxx/include/tuple:1354: decltype(auto) std::__apply_tuple_impl[abi:ne190107]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1&, std::tuple<>&, std::__tuple_indices<...>)
 [ 5366 ] {} <Fatal> BaseDaemon: 25.3. inlined from ./contrib/llvm-project/libcxx/include/tuple:1358: decltype(auto) std::apply[abi:ne190107]<DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1&, std::tuple<>&>(DB::BackgroundSchedulePool::BackgroundSchedulePool(unsigned long, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, StrongTypedef<unsigned long, CurrentMetrics::MetricTag>, char const*)::$_1&, std::tuple<>&)
 [ 5366 ] {} <Fatal> BaseDaemon: 25.4. inlined from ./src/Common/ThreadPool.h:312: operator()

Changelog category (leave one):

Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

...

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

clickhouse-gh · 2025-07-02T09:09:04Z

Workflow [PR], commit [f6af8b4]

Summary: ❌

job_name	test_name	status
Stateless tests (amd_binary, old analyzer, s3 storage, DatabaseReplicated, 2/2)		failure
	Logical error thrown (see clickhouse-server.log or logical_errors.txt)	FAIL
Stress test (amd_tsan)		failure
	Sanitizer assert (in stderr.log)	FAIL
	Fatal message in clickhouse-server.log (see fatal_messages.txt)	FAIL
	Found signal in gdb.log	FAIL

CheSema · 2025-07-02T12:15:32Z

Stress test (amd_tsan)
ThreadSanitizer: thread leak
#83063

Stateless tests (amd_binary, old analyzer, s3 storage, DatabaseReplicated, 2/2)
Lock is lost, node does not exist
#74734

clickhouse-gh Bot added the pr-not-for-changelog This PR should not be mentioned in the changelog label Jul 2, 2025

fix read from broken-connection

f6af8b4

CheSema force-pushed the chesema-fix-read-from-broken-connection branch from ce14b70 to f6af8b4 Compare July 2, 2025 09:09

Michicosun approved these changes Jul 2, 2025

View reviewed changes

Michicosun self-assigned this Jul 2, 2025

CheSema mentioned this pull request Jul 2, 2025

fix tcp connection polls, reconnect broken connections #83059

Merged

1 task

CheSema enabled auto-merge July 2, 2025 09:29

CheSema changed the title ~~fix read from broken-connection~~ disconnect broken-connection Jul 2, 2025

CheSema added this pull request to the merge queue Jul 3, 2025

Merged via the queue into master with commit d3586d8 Jul 3, 2025
118 of 123 checks passed

CheSema deleted the chesema-fix-read-from-broken-connection branch July 3, 2025 10:57

robot-ch-test-poll3 added the pr-synced-to-cloud The PR is synced to the cloud repo label Jul 3, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

disconnect broken-connection#83056

disconnect broken-connection#83056
CheSema merged 1 commit intomasterfrom
chesema-fix-read-from-broken-connection

CheSema commented Jul 2, 2025 •

edited

Loading

Uh oh!

clickhouse-gh Bot commented Jul 2, 2025 •

edited

Loading

Uh oh!

CheSema commented Jul 2, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

CheSema commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

Uh oh!

clickhouse-gh Bot commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CheSema commented Jul 2, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CheSema commented Jul 2, 2025 •

edited

Loading

clickhouse-gh Bot commented Jul 2, 2025 •

edited

Loading