Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lock-order-inversion in RemoteQueryExecutor::sendQuery #48534

Closed
kssenii opened this issue Apr 7, 2023 · 8 comments · Fixed by #48670
Closed

lock-order-inversion in RemoteQueryExecutor::sendQuery #48534

kssenii opened this issue Apr 7, 2023 · 8 comments · Fixed by #48670
Assignees
@Avogar
Labels
testing Special issue with list of bugs found by CI

Comments

@kssenii
Copy link
Member

kssenii commented Apr 7, 2023

https://s3.amazonaws.com/clickhouse-test-reports/48311/2d335f82cc8ea3dc18f6d05778de1b19c63aec01/stateless_tests__tsan__[3/5]/stderr.log

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=674)
  Cycle in lock order graph: M0 (0x7f2dbc96b3a0) => M1 (0x7f2dbc96b3d0) => M0

  Mutex M1 acquired here while holding mutex M0 in thread T563:
    #0 pthread_mutex_lock <null> (clickhouse+0xc656a5a) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #1 std::__1::__libcpp_mutex_lock[abi:v15000](pthread_mutex_t*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:304:10 (clickhouse+0x273dae39) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #2 std::__1::mutex::lock() build_docker/./contrib/llvm-project/libcxx/src/mutex.cpp:38:14 (clickhouse+0x273dae39)
    #3 std::__1::lock_guard<std::__1::mutex>::lock_guard[abi:v15000](std::__1::mutex&) build_docker/./contrib/llvm-project/libcxx/include/__mutex_base:94:27 (clickhouse+0x1cb6ac2d) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #4 DB::RemoteQueryExecutor::sendQuery(DB::ClientInfo::QueryKind) build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:240:25 (clickhouse+0x1cb6ac2d)
    #5 DB::RemoteQueryExecutor::read() build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:272:9 (clickhouse+0x1cb6c4d0) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #6 DB::RemoteQueryExecutor::readBlock() build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:260:20 (clickhouse+0x1cb6c296) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #7 DB::getStructureOfRemoteTableInShard(DB::Cluster const&, DB::Cluster::ShardInfo const&, DB::StorageID const&, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST> const&) build_docker/./src/Storages/getStructureOfRemoteTable.cpp:82:37 (clickhouse+0x1ee389fb) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #8 DB::getStructureOfRemoteTable(DB::Cluster const&, DB::StorageID const&, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST> const&) build_docker/./src/Storages/getStructureOfRemoteTable.cpp:141:32 (clickhouse+0x1ee3acd7) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #9 DB::TableFunctionRemote::getActualTableStructure(std::__1::shared_ptr<DB::Context const>) const build_docker/./src/TableFunctions/TableFunctionRemote.cpp:321:12 (clickhouse+0x1c05c03c) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #10 DB::TableFunctionRemote::executeImpl(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::ColumnsDescription) const build_docker/./src/TableFunctions/TableFunctionRemote.cpp:281:26 (clickhouse+0x1c05ae53) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #11 DB::ITableFunction::execute(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::ColumnsDescription, bool, bool) const build_docker/./src/TableFunctions/ITableFunction.cpp:37:16 (clickhouse+0x1c4a1df2) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #12 DB::Context::executeTableFunction(std::__1::shared_ptr<DB::IAST> const&, DB::ASTSelectQuery const*) build_docker/./src/Interpreters/Context.cpp:1502:35 (clickhouse+0x1d0840be) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #13 DB::JoinedTables::getLeftTableStorage() build_docker/./src/Interpreters/JoinedTables.cpp:211:44 (clickhouse+0x1e0b88c9) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #14 DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context> const&, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::shared_ptr<DB::PreparedSets>) build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:423:33 (clickhouse+0x1df9c554) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #15 DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:202:7 (clickhouse+0x1df9ad25) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #16 std::__1::__unique_if<DB::InterpreterSelectQuery>::__unique_single std::__1::make_unique[abi:v15000]<DB::InterpreterSelectQuery, std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&>(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 (clickhouse+0x1e0666fe) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #17 DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::__1::shared_ptr<DB::IAST> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:254:16 (clickhouse+0x1e0666fe)
    #18 DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:152:13 (clickhouse+0x1e0639a3) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #19 DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:41:7 (clickhouse+0x1e067959) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #20 DB::InterpreterSelectWithUnionQuery::getSampleBlock(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, bool, bool) build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:286:25 (clickhouse+0x1e067959)
    #21 DB::getColumnsFromTableExpression(DB::ASTTableExpression const&, std::__1::shared_ptr<DB::Context const>, DB::NamesAndTypesList&, DB::NamesAndTypesList&, DB::NamesAndTypesList&, bool) build_docker/./src/Interpreters/getTableExpressions.cpp:90:31 (clickhouse+0x1e48c8f9) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #22 DB::getDatabaseAndTablesWithColumns(std::__1::vector<DB::ASTTableExpression const*, std::__1::allocator<DB::ASTTableExpression const*>> const&, std::__1::shared_ptr<DB::Context const>, bool, bool, bool) build_docker/./src/Interpreters/getTableExpressions.cpp:135:45 (clickhouse+0x1e48c8f9)
    #23 DB::JoinedTables::resolveTables() build_docker/./src/Interpreters/JoinedTables.cpp:243:27 (clickhouse+0x1e0b8fcf) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #24 DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context> const&, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::shared_ptr<DB::PreparedSets>) build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:439:37 (clickhouse+0x1df9bedf) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #25 DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./src/Interpreters/InterpreterSelectQuery.cpp:202:7 (clickhouse+0x1df9ad25) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #26 std::__1::__unique_if<DB::InterpreterSelectQuery>::__unique_single std::__1::make_unique[abi:v15000]<DB::InterpreterSelectQuery, std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&>(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context>&, DB::SelectQueryOptions&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 (clickhouse+0x1e0666fe) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #27 DB::InterpreterSelectWithUnionQuery::buildCurrentChildInterpreter(std::__1::shared_ptr<DB::IAST> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:254:16 (clickhouse+0x1e0666fe)
    #28 DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context>, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&) build_docker/./src/Interpreters/InterpreterSelectWithUnionQuery.cpp:152:13 (clickhouse+0x1e0639a3) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #29 std::__1::__unique_if<DB::InterpreterSelectWithUnionQuery>::__unique_single std::__1::make_unique[abi:v15000]<DB::InterpreterSelectWithUnionQuery, std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 (clickhouse+0x1df4d848) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #30 DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) build_docker/./src/Interpreters/InterpreterFactory.cpp:148:16 (clickhouse+0x1df4c7fd) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #31 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:663:27 (clickhouse+0x1e467ef6) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #32 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1166:30 (clickhouse+0x1e465bcb) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #33 DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:420:24 (clickhouse+0x1f7bebea) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #34 DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2038:9 (clickhouse+0x1f7d49e7) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #35 Poco::Net::TCPServerConnection::start() build_docker/./base/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x23576722) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #36 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115:20 (clickhouse+0x23576f92) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #37 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:188:14 (clickhouse+0x237d13b9) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #38 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x237cf76f) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #39 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x237cdd87) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M0 acquired here while holding mutex M1 in thread T563:
    #0 pthread_mutex_lock <null> (clickhouse+0xc656a5a) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #1 std::__1::__libcpp_mutex_lock[abi:v15000](pthread_mutex_t*) build_docker/./contrib/llvm-project/libcxx/include/__threading_support:304:10 (clickhouse+0x273dae39) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #2 std::__1::mutex::lock() build_docker/./contrib/llvm-project/libcxx/src/mutex.cpp:38:14 (clickhouse+0x273dae39)
    #3 std::__1::lock_guard<std::__1::mutex>::lock_guard[abi:v15000](std::__1::mutex&) build_docker/./contrib/llvm-project/libcxx/include/__mutex_base:94:27 (clickhouse+0x1cb6aea1) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #4 DB::RemoteQueryExecutor::sendExternalTables() build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:589:25 (clickhouse+0x1cb6aea1)
    #5 DB::RemoteQueryExecutor::sendQuery(DB::ClientInfo::QueryKind) build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:252:5 (clickhouse+0x1cb6ad54) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #6 DB::RemoteQueryExecutor::read() build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:272:9 (clickhouse+0x1cb6c4d0) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #7 DB::RemoteQueryExecutor::readBlock() build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:260:20 (clickhouse+0x1cb6c296) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #8 DB::getStructureOfRemoteTableInShard(DB::Cluster const&, DB::Cluster::ShardInfo const&, DB::StorageID const&, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST> const&) build_docker/./src/Storages/getStructureOfRemoteTable.cpp:82:37 (clickhouse+0x1ee389fb) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #9 DB::getStructureOfRemoteTable(DB::Cluster const&, DB::StorageID const&, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST> const&) build_docker/./src/Storages/getStructureOfRemoteTable.cpp:141:32 (clickhouse+0x1ee3acd7) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #10 DB::TableFunctionRemote::getActualTableStructure(std::__1::shared_ptr<DB::Context const>) const build_docker/./src/TableFunctions/TableFunctionRemote.cpp:321:12 (clickhouse+0x1c05c03c) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #11 DB::TableFunctionRemote::executeImpl(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::ColumnsDescription) const build_docker/./src/TableFunctions/TableFunctionRemote.cpp:281:26 (clickhouse+0x1c05ae53) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #12 DB::ITableFunction::execute(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::ColumnsDescription, bool, bool) const build_docker/./src/TableFunctions/ITableFunction.cpp:37:16 (clickhouse+0x1c4a1df2) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #13 DB::(anonymous namespace)::QueryAnalyzer::resolveTableFunction(std::__1::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, DB::(anonymous namespace)::QueryExpressionsAliasVisitor&, bool) build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:6214:55 (clickhouse+0x1de651c7) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #14 DB::(anonymous namespace)::QueryAnalyzer::resolveQueryJoinTreeNode(std::__1::shared_ptr<DB::IQueryTreeNode>&, DB::(anonymous namespace)::IdentifierResolveScope&, DB::(anonymous namespace)::QueryExpressionsAliasVisitor&) build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:6446:13 (clickhouse+0x1de674b4) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #15 DB::(anonymous namespace)::QueryAnalyzer::resolveQuery(std::__1::shared_ptr<DB::IQueryTreeNode> const&, DB::(anonymous namespace)::IdentifierResolveScope&) build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:6683:9 (clickhouse+0x1de547d3) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #16 DB::(anonymous namespace)::QueryAnalyzer::resolve(std::__1::shared_ptr<DB::IQueryTreeNode>, std::__1::shared_ptr<DB::IQueryTreeNode> const&, std::__1::shared_ptr<DB::Context const>) build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:1071:17 (clickhouse+0x1de51b47) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #17 DB::QueryAnalysisPass::run(std::__1::shared_ptr<DB::IQueryTreeNode>, std::__1::shared_ptr<DB::Context const>) build_docker/./src/Analyzer/Passes/QueryAnalysisPass.cpp:6934:14 (clickhouse+0x1de51b47)
    #18 DB::QueryTreePassManager::run(std::__1::shared_ptr<DB::IQueryTreeNode>) build_docker/./src/Analyzer/QueryTreePassManager.cpp:184:20 (clickhouse+0x1de4ee7c) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #19 DB::(anonymous namespace)::buildQueryTreeAndRunPasses(std::__1::shared_ptr<DB::IAST> const&, DB::SelectQueryOptions const&, std::__1::shared_ptr<DB::Context const> const&, std::__1::shared_ptr<DB::IStorage> const&) build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:141:33 (clickhouse+0x1e008128) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #20 DB::InterpreterSelectQueryAnalyzer::InterpreterSelectQueryAnalyzer(std::__1::shared_ptr<DB::IAST> const&, std::__1::shared_ptr<DB::Context const> const&, DB::SelectQueryOptions const&) build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:158:18 (clickhouse+0x1e006a60) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #21 std::__1::__unique_if<DB::InterpreterSelectQueryAnalyzer>::__unique_single std::__1::make_unique[abi:v15000]<DB::InterpreterSelectQueryAnalyzer, std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&>(std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>&, DB::SelectQueryOptions const&) build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:714:32 (clickhouse+0x1df4c790) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #22 DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, std::__1::shared_ptr<DB::Context>, DB::SelectQueryOptions const&) build_docker/./src/Interpreters/InterpreterFactory.cpp:146:20 (clickhouse+0x1df4c790)
    #23 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:663:27 (clickhouse+0x1e467ef6) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #24 DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, std::__1::shared_ptr<DB::Context>, std::__1::function<void (DB::QueryResultDetails const&)>, std::__1::optional<DB::FormatSettings> const&) build_docker/./src/Interpreters/executeQuery.cpp:1260:30 (clickhouse+0x1e46f38f) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #25 DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::__1::optional<DB::CurrentThread::QueryScope>&) build_docker/./src/Server/HTTPHandler.cpp:838:5 (clickhouse+0x1f753db6) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #26 DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&) build_docker/./src/Server/HTTPHandler.cpp:1033:9 (clickhouse+0x1f7584b0) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #27 DB::HTTPServerConnection::run() build_docker/./src/Server/HTTP/HTTPServerConnection.cpp:68:34 (clickhouse+0x1f7e1683) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #28 Poco::Net::TCPServerConnection::start() build_docker/./base/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x23576722) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #29 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115:20 (clickhouse+0x23576f92) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #30 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:188:14 (clickhouse+0x237d13b9) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #31 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x237cf76f) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #32 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x237cdd87) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)

  Thread T563 'HTTPHandler' (tid=3513, running) created by thread T326 at:
    #0 pthread_create <null> (clickhouse+0xc638d7d) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #1 Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable>>) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:198:6 (clickhouse+0x237cd793) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #2 Poco::Thread::start(Poco::Runnable&) build_docker/./base/poco/Foundation/src/Thread.cpp:118:2 (clickhouse+0x237cf1ac) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #3 Poco::PooledThread::start() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:78:10 (clickhouse+0x237d330b) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #4 Poco::ThreadPool::getThread() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:446:14 (clickhouse+0x237d330b)
    #5 Poco::ThreadPool::startWithPriority(Poco::Thread::Priority, Poco::Runnable&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/./base/poco/Foundation/src/ThreadPool.cpp:350:2 (clickhouse+0x237d3707) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #6 Poco::Net::TCPServerDispatcher::enqueue(Poco::Net::StreamSocket const&) build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:152:17 (clickhouse+0x235774a6) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #7 Poco::Net::TCPServer::run() build_docker/./base/poco/Net/src/TCPServer.cpp:148:21 (clickhouse+0x23576007) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #8 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x237cf76f) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)
    #9 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x237cdd87) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) (/usr/bin/clickhouse+0xc656a5a) (BuildId: 3eb266499adf7a64fbb871410d99cc36a98b18a3) in pthread_mutex_lock
@kssenii kssenii added the testing Special issue with list of bugs found by CI label Apr 7, 2023
@kssenii
Copy link
Member Author

kssenii commented Apr 7, 2023

cc @Avogar

@kssenii kssenii mentioned this issue Apr 7, 2023
Merged
@azat
Copy link
Collaborator

azat commented Apr 10, 2023

One more - https://s3.amazonaws.com/clickhouse-test-reports/48570/cd7e45128c514d8b496d51325a4af11d66d1f530/stateless_tests__tsan__[3/5].html

@Avogar Avogar self-assigned this Apr 11, 2023
@Avogar
Copy link
Member

Avogar commented Apr 11, 2023

Hmm. I don't understand. As I can see from the first stacktrace, M1 is located in RemoteQueryExecutor.cpp:240, so it's duplicated_part_uuids_mutex. And from the second stacktrace, M0 is located in RemoteQueryExecutor.cpp:589, so it's external_tables_mutex. But in the first stacktrace there is no place where we could lock external_tables_mutex and in the second stacktrace there is no place where we could lock duplicated_part_uuids_mutex...
I don't understand how thread sanitizer decided that we lock duplicated_part_uuids_mutex while holding external_tables_mutex or vice versa. I just don't see any possibility when these two mutexes can be locked in one stacktrace.
Both mutexts can be loked while holding another mutex was_cancelled_mutex but it's ok (and this report doesn't say anything about this mutex)

@yakov-olkhovskiy yakov-olkhovskiy mentioned this issue Apr 11, 2023
Merged
@KochetovNicolai
Copy link
Member

https://s3.amazonaws.com/clickhouse-test-reports/48611/7581982e0a6af04ad953bb68ae151fefdfc1593f/stateless_tests__tsan__[3/5].html

@tavplubix tavplubix mentioned this issue Apr 11, 2023
Merged
@tavplubix
Copy link
Member

It can be a result of a memory safety issue. For example, we tried to lock a mutex that was already destroyed.

@Avogar
Copy link
Member

Avogar commented Apr 11, 2023 •
edited

Also Nikolai mentioned interesting thing, both stacktraces have the same thread T563

@Avogar
Copy link
Member

Avogar commented Apr 11, 2023 •
edited

Probably it's from test 01455_opentelemetry_distributed.sh where we have query select 1 from remote('127.0.0.2', system, one) settings allow_experimental_analyzer = 1 format Null. I tried run this test in parallel many times locally with tsan build, cannot reproduce it...
Maybe there are some changed settings, but I cannot find them

azat added a commit to azat/ClickHouse that referenced this issue Apr 11, 2023
@azat
Remove lock for duplicated parts UUIDs (allow_experimental_query_dedu…
d7f8650 
…plication=1)

It looks redundant, since sendQuery() cannot be executed in parallel
with processPacket() (hence RemoteQueryExecutor::setPartUUIDs())

This likely will fix the lock-order-inversion in RemoteQueryExecutor,
since I think it is false-positive.

Fixes: ClickHouse#48534
Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
@azat azat mentioned this issue Apr 11, 2023
Merged
@azat
Copy link
Collaborator

azat commented Apr 11, 2023

I think that this one is false-positive, maybe google/sanitizers#814 looks similar.

But actually looking into this code, I don't see that duplicated_part_uuids_mutex is required, so how about #48670 ?

Other thing that I have in mind is that external_tables_mutex can be replaced with was_cancelled_mutex since later guards the whole sendQuery (#17254).

But, even better will be to remove was_cancelled_mutex from the sendQuery this will allow to cancel the query earlier, but I need to play with the code for this (I need to reproduce the problem that #17254 solves, so far no results)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Avogar Avogar

Labels
testing Special issue with list of bugs found by CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove lock for duplicated parts UUIDs (allow_experimental_query_deduplication=1) azat/ClickHouse
5 participants
@azat @KochetovNicolai @tavplubix @Avogar @kssenii