Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Parallel replicas - experimental feature, not ready for production) Logical error: '!is_parallel_reading_from_replicas' #57716

Closed
vitlibar opened this issue Dec 10, 2023 · 2 comments
Assignees
Labels
comp-parallel-replicas Parallel reading from replicas experimental feature Bug in the feature that should not be used in production fuzz Problem found by one of the fuzzers

Comments

@vitlibar
Copy link
Member

vitlibar commented Dec 10, 2023

Found by AST fuzzer (ubsan):

2023.12.09 21:16:09.298619 [ 1188 ] {c8859fe5-58da-4122-a5d7-e583dd98112a} <Fatal> : Logical error: '!is_parallel_reading_from_replicas'.

2023.12.09 21:16:09.299398 [ 1191 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007f2424dee9fc 0x00007f2424d9a476 0x00007f2424d807f3 0x000055cc0a011aa8 0x000055cc197b9668 0x000055cc197bc107 0x000055cc19752e85 0x000055cc19783b41 0x000055cc16d9735e 0x000055cc16d96dfc 0x000055cc17410ae9 0x000055cc17409887 0x000055cc18d4b266 0x000055cc18d7b5b2 0x000055cc1a98f11e 0x000055cc1a99025a 0x000055cc1abf70b0 0x000055cc1abf21b1 0x00007f2424decac3 0x00007f2424e7ea40
2023.12.09 21:16:09.299408 [ 1191 ] {} <Fatal> BaseDaemon: 3. ? @ 0x00007f2424dee9fc in ?
2023.12.09 21:16:09.299415 [ 1191 ] {} <Fatal> BaseDaemon: 4. ? @ 0x00007f2424d9a476 in ?
2023.12.09 21:16:09.299422 [ 1191 ] {} <Fatal> BaseDaemon: 5. ? @ 0x00007f2424d807f3 in ?
2023.12.09 21:16:09.312597 [ 1191 ] {} <Fatal> BaseDaemon: 6. ./build_docker/./src/Common/Exception.cpp:44: DB::abortOnFailedAssertion(String const&) @ 0x0000000023529aa8 in /workspace/clickhouse
2023.12.09 21:16:09.312596 [ 1190 ] {} <Fatal> BaseDaemon: 6. ./build_docker/./src/Common/Exception.cpp:44: DB::abortOnFailedAssertion(String const&) @ 0x0000000023529aa8 in /workspace/clickhouse
2023.12.09 21:16:09.312601 [ 1189 ] {} <Fatal> BaseDaemon: 6. ./build_docker/./src/Common/Exception.cpp:44: DB::abortOnFailedAssertion(String const&) @ 0x0000000023529aa8 in /workspace/clickhouse
2023.12.09 21:16:09.382616 [ 1191 ] {} <Fatal> BaseDaemon: 7. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1825: DB::ReadFromMergeTree::spreadMarkRanges(DB::RangesInDataParts&&, unsigned long, DB::ReadFromMergeTree::AnalysisResult&, std::shared_ptr<DB::ActionsDAG>&) @ 0x0000000032cd1668 in /workspace/clickhouse
2023.12.09 21:16:09.382686 [ 1190 ] {} <Fatal> BaseDaemon: 7. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1825: DB::ReadFromMergeTree::spreadMarkRanges(DB::RangesInDataParts&&, unsigned long, DB::ReadFromMergeTree::AnalysisResult&, std::shared_ptr<DB::ActionsDAG>&) @ 0x0000000032cd1668 in /workspace/clickhouse
2023.12.09 21:16:09.382729 [ 1189 ] {} <Fatal> BaseDaemon: 7. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1825: DB::ReadFromMergeTree::spreadMarkRanges(DB::RangesInDataParts&&, unsigned long, DB::ReadFromMergeTree::AnalysisResult&, std::shared_ptr<DB::ActionsDAG>&) @ 0x0000000032cd1668 in /workspace/clickhouse
2023.12.09 21:16:09.439902 [ 1191 ] {} <Fatal> BaseDaemon: 8.1. inlined from ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:807: std::shared_ptr<std::vector<std::shared_ptr<DB::IProcessor>, std::allocator<std::shared_ptr<DB::IProcessor>>>>::operator*[abi:v15000]() const
2023.12.09 21:16:09.439933 [ 1191 ] {} <Fatal> BaseDaemon: 8.2. inlined from ./src/QueryPipeline/Pipe.h:103: DB::Pipe::getProcessors() const
2023.12.09 21:16:09.439944 [ 1191 ] {} <Fatal> BaseDaemon: 8. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1946: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000032cd4107 in /workspace/clickhouse
2023.12.09 21:16:09.440095 [ 1190 ] {} <Fatal> BaseDaemon: 8.1. inlined from ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:807: std::shared_ptr<std::vector<std::shared_ptr<DB::IProcessor>, std::allocator<std::shared_ptr<DB::IProcessor>>>>::operator*[abi:v15000]() const
2023.12.09 21:16:09.440124 [ 1190 ] {} <Fatal> BaseDaemon: 8.2. inlined from ./src/QueryPipeline/Pipe.h:103: DB::Pipe::getProcessors() const
2023.12.09 21:16:09.440133 [ 1190 ] {} <Fatal> BaseDaemon: 8. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1946: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000032cd4107 in /workspace/clickhouse
2023.12.09 21:16:09.440226 [ 1189 ] {} <Fatal> BaseDaemon: 8.1. inlined from ./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:807: std::shared_ptr<std::vector<std::shared_ptr<DB::IProcessor>, std::allocator<std::shared_ptr<DB::IProcessor>>>>::operator*[abi:v15000]() const
2023.12.09 21:16:09.440261 [ 1189 ] {} <Fatal> BaseDaemon: 8.2. inlined from ./src/QueryPipeline/Pipe.h:103: DB::Pipe::getProcessors() const
2023.12.09 21:16:09.440270 [ 1189 ] {} <Fatal> BaseDaemon: 8. ./build_docker/./src/Processors/QueryPlan/ReadFromMergeTree.cpp:1946: DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x0000000032cd4107 in /workspace/clickhouse
2023.12.09 21:16:09.447489 [ 1191 ] {} <Fatal> BaseDaemon: 9. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000032c6ae85 in /workspace/clickhouse
2023.12.09 21:16:09.447548 [ 1190 ] {} <Fatal> BaseDaemon: 9. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000032c6ae85 in /workspace/clickhouse
2023.12.09 21:16:09.447740 [ 1189 ] {} <Fatal> BaseDaemon: 9. ./build_docker/./src/Processors/QueryPlan/ISourceStep.cpp:24: DB::ISourceStep::updatePipeline(std::vector<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>, std::allocator<std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>>>, DB::BuildQueryPipelineSettings const&) @ 0x0000000032c6ae85 in /workspace/clickhouse
2023.12.09 21:16:09.468380 [ 1191 ] {} <Fatal> BaseDaemon: 10.1. inlined from ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>::reset[abi:v15000](DB::QueryPipelineBuilder*)
2023.12.09 21:16:09.468408 [ 1191 ] {} <Fatal> BaseDaemon: 10.2. inlined from ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:225: std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>::operator=[abi:v15000](std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>&&)
2023.12.09 21:16:09.468421 [ 1191 ] {} <Fatal> BaseDaemon: 10. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000032c9bb41 in /workspace/clickhouse
2023.12.09 21:16:09.468496 [ 1189 ] {} <Fatal> BaseDaemon: 10.1. inlined from ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>::reset[abi:v15000](DB::QueryPipelineBuilder*)
2023.12.09 21:16:09.468522 [ 1189 ] {} <Fatal> BaseDaemon: 10.2. inlined from ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:225: std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>::operator=[abi:v15000](std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>&&)
2023.12.09 21:16:09.468533 [ 1189 ] {} <Fatal> BaseDaemon: 10. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000032c9bb41 in /workspace/clickhouse
2023.12.09 21:16:09.468532 [ 1190 ] {} <Fatal> BaseDaemon: 10.1. inlined from ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>::reset[abi:v15000](DB::QueryPipelineBuilder*)
2023.12.09 21:16:09.468566 [ 1190 ] {} <Fatal> BaseDaemon: 10.2. inlined from ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:225: std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>::operator=[abi:v15000](std::unique_ptr<DB::QueryPipelineBuilder, std::default_delete<DB::QueryPipelineBuilder>>&&)
2023.12.09 21:16:09.468576 [ 1190 ] {} <Fatal> BaseDaemon: 10. ./build_docker/./src/Processors/QueryPlan/QueryPlan.cpp:188: DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x0000000032c9bb41 in /workspace/clickhouse
2023.12.09 21:16:09.482299 [ 1190 ] {} <Fatal> BaseDaemon: 11. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:0: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x00000000302af35e in /workspace/clickhouse
2023.12.09 21:16:09.482308 [ 1191 ] {} <Fatal> BaseDaemon: 11. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:0: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x00000000302af35e in /workspace/clickhouse
2023.12.09 21:16:09.482301 [ 1189 ] {} <Fatal> BaseDaemon: 11. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:0: DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x00000000302af35e in /workspace/clickhouse
2023.12.09 21:16:09.495760 [ 1189 ] {} <Fatal> BaseDaemon: 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:225: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x00000000302aedfc in /workspace/clickhouse
2023.12.09 21:16:09.495791 [ 1191 ] {} <Fatal> BaseDaemon: 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:225: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x00000000302aedfc in /workspace/clickhouse
2023.12.09 21:16:09.496798 [ 1190 ] {} <Fatal> BaseDaemon: 12. ./build_docker/./src/Interpreters/InterpreterSelectQueryAnalyzer.cpp:225: DB::InterpreterSelectQueryAnalyzer::execute() @ 0x00000000302aedfc in /workspace/clickhouse
2023.12.09 21:16:09.534783 [ 1189 ] {} <Fatal> BaseDaemon: 13. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000030928ae9 in /workspace/clickhouse
2023.12.09 21:16:09.534851 [ 1191 ] {} <Fatal> BaseDaemon: 13. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000030928ae9 in /workspace/clickhouse
2023.12.09 21:16:09.536139 [ 1190 ] {} <Fatal> BaseDaemon: 13. ./build_docker/./src/Interpreters/executeQuery.cpp:0: DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000030928ae9 in /workspace/clickhouse
2023.12.09 21:16:09.577844 [ 1191 ] {} <Fatal> BaseDaemon: 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1287: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000030921887 in /workspace/clickhouse
2023.12.09 21:16:09.577886 [ 1189 ] {} <Fatal> BaseDaemon: 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1287: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000030921887 in /workspace/clickhouse
2023.12.09 21:16:09.579696 [ 1190 ] {} <Fatal> BaseDaemon: 14. ./build_docker/./src/Interpreters/executeQuery.cpp:1287: DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x0000000030921887 in /workspace/clickhouse
2023.12.09 21:16:09.612122 [ 1189 ] {} <Fatal> BaseDaemon: 15. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000032263266 in /workspace/clickhouse
2023.12.09 21:16:09.612233 [ 1191 ] {} <Fatal> BaseDaemon: 15. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000032263266 in /workspace/clickhouse
2023.12.09 21:16:09.613049 [ 1190 ] {} <Fatal> BaseDaemon: 15. ./build_docker/./src/Server/TCPHandler.cpp:0: DB::TCPHandler::runImpl() @ 0x0000000032263266 in /workspace/clickhouse
2023.12.09 21:16:09.628463 [ 559 ] {} <Trace> system.asynchronous_metric_log_sender.DistributedInsertQueue.default: Finished processing `/workspace/db/store/5a7/5a74ea5a-2981-4d34-9b7b-76bf00071867/shard1_replica1/82.bin` (took 2424 ms)
2023.12.09 21:16:09.628502 [ 559 ] {} <Trace> system.asynchronous_metric_log_sender.DistributedInsertQueue.default/Bg: Execution took 2424 ms.
2023.12.09 21:16:09.649875 [ 643 ] {} <Trace> system.text_log_sender.DistributedInsertQueue.default: Finished processing `/workspace/db/store/017/0171101c-75fa-4cdb-aa56-2aa647f58554/shard1_replica1/131.bin` (took 588 ms)
2023.12.09 21:16:09.649909 [ 643 ] {} <Trace> system.text_log_sender.DistributedInsertQueue.default/Bg: Execution took 588 ms.
2023.12.09 21:16:09.662827 [ 1189 ] {} <Fatal> BaseDaemon: 16. ./build_docker/./src/Server/TCPHandler.cpp:2294: DB::TCPHandler::run() @ 0x00000000322935b2 in /workspace/clickhouse
2023.12.09 21:16:09.663023 [ 1191 ] {} <Fatal> BaseDaemon: 16. ./build_docker/./src/Server/TCPHandler.cpp:2294: DB::TCPHandler::run() @ 0x00000000322935b2 in /workspace/clickhouse
2023.12.09 21:16:09.664307 [ 1190 ] {} <Fatal> BaseDaemon: 16. ./build_docker/./src/Server/TCPHandler.cpp:2294: DB::TCPHandler::run() @ 0x00000000322935b2 in /workspace/clickhouse
2023.12.09 21:16:09.665447 [ 1189 ] {} <Fatal> BaseDaemon: 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000033ea711e in /workspace/clickhouse
2023.12.09 21:16:09.665627 [ 1191 ] {} <Fatal> BaseDaemon: 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000033ea711e in /workspace/clickhouse
2023.12.09 21:16:09.666940 [ 1190 ] {} <Fatal> BaseDaemon: 17. ./build_docker/./base/poco/Net/src/TCPServerConnection.cpp:57: Poco::Net::TCPServerConnection::start() @ 0x0000000033ea711e in /workspace/clickhouse
2023.12.09 21:16:09.669140 [ 1189 ] {} <Fatal> BaseDaemon: 18.1. inlined from ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: std::unique_ptr<Poco::Net::TCPServerConnection, std::default_delete<Poco::Net::TCPServerConnection>>::reset[abi:v15000](Poco::Net::TCPServerConnection*)
2023.12.09 21:16:09.669149 [ 1191 ] {} <Fatal> BaseDaemon: 18.1. inlined from ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: std::unique_ptr<Poco::Net::TCPServerConnection, std::default_delete<Poco::Net::TCPServerConnection>>::reset[abi:v15000](Poco::Net::TCPServerConnection*)
2023.12.09 21:16:09.669163 [ 1189 ] {} <Fatal> BaseDaemon: 18.2. inlined from ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259: ~unique_ptr
2023.12.09 21:16:09.669172 [ 1191 ] {} <Fatal> BaseDaemon: 18.2. inlined from ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259: ~unique_ptr
2023.12.09 21:16:09.669172 [ 1189 ] {} <Fatal> BaseDaemon: 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:116: Poco::Net::TCPServerDispatcher::run() @ 0x0000000033ea825a in /workspace/clickhouse
2023.12.09 21:16:09.669183 [ 1191 ] {} <Fatal> BaseDaemon: 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:116: Poco::Net::TCPServerDispatcher::run() @ 0x0000000033ea825a in /workspace/clickhouse
2023.12.09 21:16:09.670360 [ 1190 ] {} <Fatal> BaseDaemon: 18.1. inlined from ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: std::unique_ptr<Poco::Net::TCPServerConnection, std::default_delete<Poco::Net::TCPServerConnection>>::reset[abi:v15000](Poco::Net::TCPServerConnection*)
2023.12.09 21:16:09.670381 [ 1190 ] {} <Fatal> BaseDaemon: 18.2. inlined from ./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:259: ~unique_ptr
2023.12.09 21:16:09.670391 [ 1190 ] {} <Fatal> BaseDaemon: 18. ./build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:116: Poco::Net::TCPServerDispatcher::run() @ 0x0000000033ea825a in /workspace/clickhouse

See other failures

The failure happened in this chassert:

        chassert(!is_parallel_reading_from_replicas);
@vitlibar vitlibar added the fuzz Problem found by one of the fuzzers label Dec 10, 2023
@alexey-milovidov alexey-milovidov added the experimental feature Bug in the feature that should not be used in production label Dec 10, 2023
@alexey-milovidov alexey-milovidov changed the title Logical error: '!is_parallel_reading_from_replicas' (Parallel replicas - experimental feature, not ready for production) Logical error: '!is_parallel_reading_from_replicas' Dec 10, 2023
@Algunenano Algunenano self-assigned this Dec 11, 2023
@Algunenano
Copy link
Member

Should be fixed by #57414. The problem is FINAL + parallel replicas, which isn't supported but it wasn't disabled with the analyzer.

@Algunenano Algunenano added the comp-parallel-replicas Parallel reading from replicas label Dec 11, 2023
@devcrafter
Copy link
Member

#57414 merged, let's close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp-parallel-replicas Parallel reading from replicas experimental feature Bug in the feature that should not be used in production fuzz Problem found by one of the fuzzers
Projects
None yet
Development

No branches or pull requests

4 participants