Fuzzing 02950_part_offset_as_primary_key.sql with parallel replicas

[devcrafter]

Describe the bug

Query fails with

<Fatal> : Logical error: 'Replica 1 decided to read in WithOrder mode, not in Default. This is a bug'.

https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=87644&sha=51ec319faacede1a0f7864baf9085d0a6dbd8867&name_0=PR

How to reproduce

Run 02950_part_offset_as_primary_key with enabled parallel replicas

Error message and/or stacktrace

clickhouse-server.err.log:2025.09.25 19:42:53.381502 [ 83184 ] {28c04ffb-3819-4b98-a761-a55ae944ce14} <Fatal> : Logical error: 'Replica 1 decided to read in WithOrder mode, not in Default. This is a bug'.
clickhouse-server.err.log:2025.09.25 19:42:53.391921 [ 83184 ] {28c04ffb-3819-4b98-a761-a55ae944ce14} <Fatal> : Stack trace (when copying this message, always include the lines below):
clickhouse-server.err.log:2025.09.25 19:42:53.392118 [ 1622 ] {} <Fatal> BaseDaemon: ########## Short fault info ############
clickhouse-server.err.log:2025.09.25 19:42:53.392135 [ 1622 ] {} <Fatal> BaseDaemon: (version 25.10.1.692 (official build), build id: C27877DCE6D8A529E405E86B2B0172EA06EB8F42, git hash: 89a7b88905dddd20d7fe35897c0f26e4218b57e7, architecture: x86_64) (from thread 83184) Received signal 6
clickhouse-server.err.log:2025.09.25 19:42:53.392140 [ 1622 ] {} <Fatal> BaseDaemon: Signal description: Aborted
clickhouse-server.err.log:2025.09.25 19:42:53.392142 [ 1622 ] {} <Fatal> BaseDaemon: 
clickhouse-server.err.log:2025.09.25 19:42:53.392161 [ 1622 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007f01ff5529fd 0x00007f01ff4fe476 0x00007f01ff4e47f3 0x000055b5adeca153 0x000055b5adeca9ac 0x000055b5adecac6c 0x000055b5a728334e 0x000055b5a7282f40 0x000055b5b792bcab 0x000055b5b791e866 0x000055b5b304f3a1 0x000055b5b304de1b 0x000055b5b304ef3e 0x000055b5b81d0fe6 0x000055b5b7e50c76 0x000055b5b7e6bd59 0x000055b5b7e5f5d2 0x000055b5b7e60a85 0x000055b5ae0105d5 0x000055b5ae015d26 0x000055b5ae00e020 0x000055b5ae013e3b 0x00007f01ff550ac3 0x00007f01ff5e2850
clickhouse-server.err.log:2025.09.25 19:42:53.392167 [ 1622 ] {} <Fatal> BaseDaemon: ########################################
clickhouse-server.err.log:2025.09.25 19:42:53.392199 [ 1622 ] {} <Fatal> BaseDaemon: (version 25.10.1.692 (official build), build id: C27877DCE6D8A529E405E86B2B0172EA06EB8F42, git hash: 89a7b88905dddd20d7fe35897c0f26e4218b57e7) (from thread 83184) (query_id: 28c04ffb-3819-4b98-a761-a55ae944ce14) (query: -- nothing to read
clickhouse-server.err.log:2025.09.25 19:42:53.392213 [ 1622 ] {} <Fatal> BaseDaemon: 
clickhouse-server.err.log:2025.09.25 19:42:53.392230 [ 1622 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007f01ff5529fd 0x00007f01ff4fe476 0x00007f01ff4e47f3 0x000055b5adeca153 0x000055b5adeca9ac 0x000055b5adecac6c 0x000055b5a728334e 0x000055b5a7282f40 0x000055b5b792bcab 0x000055b5b791e866 0x000055b5b304f3a1 0x000055b5b304de1b 0x000055b5b304ef3e 0x000055b5b81d0fe6 0x000055b5b7e50c76 0x000055b5b7e6bd59 0x000055b5b7e5f5d2 0x000055b5b7e60a85 0x000055b5ae0105d5 0x000055b5ae015d26 0x000055b5ae00e020 0x000055b5ae013e3b 0x00007f01ff550ac3 0x00007f01ff5e2850
clickhouse-server.err.log:2025.09.25 19:42:53.392279 [ 1622 ] {} <Fatal> BaseDaemon: 3. ? @ 0x00000000000969fd
clickhouse-server.err.log:2025.09.25 19:42:53.392289 [ 1622 ] {} <Fatal> BaseDaemon: 4. ? @ 0x0000000000042476
clickhouse-server.err.log:2025.09.25 19:42:53.392303 [ 1622 ] {} <Fatal> BaseDaemon: 5. ? @ 0x00000000000287f3
clickhouse-server.err.log:2025.09.25 19:42:53.401233 [ 1622 ] {} <Fatal> BaseDaemon: 6. ./ci/tmp/build/./src/Common/Exception.cpp:51: DB::abortOnFailedAssertion(String const&, void* const*, unsigned long, unsigned long) @ 0x000000000fdce153
clickhouse-server.err.log:2025.09.25 19:42:53.410609 [ 1622 ] {} <Fatal> BaseDaemon: 7. ./ci/tmp/build/./src/Common/Exception.cpp:84: DB::handle_error_code(String const&, std::basic_string_view<char, std::char_traits<char>>, int, bool, std::vector<void*, std::allocator<void*>> const&) @ 0x000000000fdce9ac
clickhouse-server.err.log:2025.09.25 19:42:53.421274 [ 1622 ] {} <Fatal> BaseDaemon: 8. ./ci/tmp/build/./src/Common/Exception.cpp:135: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000fdcec6c
clickhouse-server.err.log:2025.09.25 19:42:53.465900 [ 1622 ] {} <Fatal> BaseDaemon: 9. DB::Exception::Exception(String&&, int, String, bool) @ 0x000000000918734e
clickhouse-server.err.log:2025.09.25 19:42:53.486682 [ 1622 ] {} <Fatal> BaseDaemon: 10. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000009186f40
clickhouse-server.err.log:2025.09.25 19:42:53.517681 [ 1622 ] {} <Fatal> BaseDaemon: 11. ./src/Common/Exception.h:141: DB::Exception::Exception<unsigned long&, std::basic_string_view<char, std::char_traits<char>>, std::basic_string_view<char, std::char_traits<char>>>(int, FormatStringHelperImpl<std::type_identity<unsigned long&>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type, std::type_identity<std::basic_string_view<char, std::char_traits<char>>>::type>, unsigned long&, std::basic_string_view<char, std::char_traits<char>>&&, std::basic_string_view<char, std::char_traits<char>>&&) @ 0x000000001982fcab
clickhouse-server.err.log:2025.09.25 19:42:53.547460 [ 1622 ] {} <Fatal> BaseDaemon: 12. ./ci/tmp/build/./src/Storages/MergeTree/ParallelReplicasReadingCoordinator.cpp:1153: DB::ParallelReplicasReadingCoordinator::handleRequest(DB::ParallelReadRequest) @ 0x0000000019822866
clickhouse-server.err.log:2025.09.25 19:42:53.580278 [ 1622 ] {} <Fatal> BaseDaemon: 13. ./ci/tmp/build/./src/QueryPipeline/RemoteQueryExecutor.cpp:764: DB::RemoteQueryExecutor::processMergeTreeReadTaskRequest(DB::ParallelReadRequest) @ 0x0000000014f533a1
clickhouse-server.err.log:2025.09.25 19:42:53.610153 [ 1622 ] {} <Fatal> BaseDaemon: 14. ./ci/tmp/build/./src/QueryPipeline/RemoteQueryExecutor.cpp:637: DB::RemoteQueryExecutor::processPacket(DB::Packet) @ 0x0000000014f51e1b
clickhouse-server.err.log:2025.09.25 19:42:53.641087 [ 1622 ] {} <Fatal> BaseDaemon: 15. ./ci/tmp/build/./src/QueryPipeline/RemoteQueryExecutor.cpp:585: DB::RemoteQueryExecutor::readAsync() @ 0x0000000014f52f3e
clickhouse-server.err.log:2025.09.25 19:42:53.651022 [ 1622 ] {} <Fatal> BaseDaemon: 16. ./ci/tmp/build/./src/Processors/Sources/RemoteSource.cpp:187: DB::RemoteSource::tryGenerate() @ 0x000000001a0d4fe6
clickhouse-server.err.log:2025.09.25 19:42:53.657267 [ 1622 ] {} <Fatal> BaseDaemon: 17. ./ci/tmp/build/./src/Processors/ISource.cpp:110: DB::ISource::work() @ 0x0000000019d54c76
clickhouse-server.err.log:2025.09.25 19:42:53.660547 [ 1622 ] {} <Fatal> BaseDaemon: 18.0. inlined from ./ci/tmp/build/./src/Processors/Executors/ExecutionThreadContext.cpp:53: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*)
clickhouse-server.err.log:2025.09.25 19:42:53.660586 [ 1622 ] {} <Fatal> BaseDaemon: 18. ./ci/tmp/build/./src/Processors/Executors/ExecutionThreadContext.cpp:102: DB::ExecutionThreadContext::executeTask() @ 0x0000000019d6fd59
clickhouse-server.err.log:2025.09.25 19:42:53.672085 [ 1622 ] {} <Fatal> BaseDaemon: 19. ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:351: DB::PipelineExecutor::executeStepImpl(unsigned long, DB::IAcquiredSlot*, std::atomic<bool>*) @ 0x0000000019d635d2
clickhouse-server.err.log:2025.09.25 19:42:53.685177 [ 1622 ] {} <Fatal> BaseDaemon: 20.0. inlined from ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:279: DB::PipelineExecutor::executeSingleThread(unsigned long, DB::IAcquiredSlot*)
clickhouse-server.err.log:2025.09.25 19:42:53.685203 [ 1622 ] {} <Fatal> BaseDaemon: 20.1. inlined from ./ci/tmp/build/./src/Processors/Executors/PipelineExecutor.cpp:565: operator()
clickhouse-server.err.log:2025.09.25 19:42:53.685226 [ 1622 ] {} <Fatal> BaseDaemon: 20.2. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:149: decltype(std::declval<DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&>()()) std::__invoke[abi:ne190107]<DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&>(DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&)
clickhouse-server.err.log:2025.09.25 19:42:53.685240 [ 1622 ] {} <Fatal> BaseDaemon: 20.3. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:224: void std::__invoke_void_return_wrapper<void, true>::__call[abi:ne190107]<DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&>(DB::PipelineExecutor::spawnThreads(std::shared_ptr<DB::IAcquiredSlot>)::$_0&)
clickhouse-server.err.log:2025.09.25 19:42:53.685248 [ 1622 ] {} <Fatal> BaseDaemon: 20.4. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:210: ?
clickhouse-server.err.log:2025.09.25 19:42:53.685253 [ 1622 ] {} <Fatal> BaseDaemon: 20. ./contrib/llvm-project/libcxx/include/__functional/function.h:610: ? @ 0x0000000019d64a85
clickhouse-server.err.log:2025.09.25 19:42:53.694856 [ 1622 ] {} <Fatal> BaseDaemon: 21.0. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ?
clickhouse-server.err.log:2025.09.25 19:42:53.694885 [ 1622 ] {} <Fatal> BaseDaemon: 21.1. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:989: ?
clickhouse-server.err.log:2025.09.25 19:42:53.694908 [ 1622 ] {} <Fatal> BaseDaemon: 21. ./ci/tmp/build/./src/Common/ThreadPool.cpp:802: ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::worker() @ 0x000000000ff145d5
clickhouse-server.err.log:2025.09.25 19:42:53.701992 [ 1622 ] {} <Fatal> BaseDaemon: 22.0. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: decltype(*std::declval<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&>().*std::declval<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)()>()()) std::__invoke[abi:ne190107]<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&, void>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&)
clickhouse-server.err.log:2025.09.25 19:42:53.702018 [ 1622 ] {} <Fatal> BaseDaemon: 22.1. inlined from ./contrib/llvm-project/libcxx/include/tuple:1354: decltype(auto) std::__apply_tuple_impl[abi:ne190107]<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), std::tuple<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>&, 0ul>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), std::tuple<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>&, std::__tuple_indices<0ul>)
clickhouse-server.err.log:2025.09.25 19:42:53.702034 [ 1622 ] {} <Fatal> BaseDaemon: 22.2. inlined from ./contrib/llvm-project/libcxx/include/tuple:1358: decltype(auto) std::apply[abi:ne190107]<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), std::tuple<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>&>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&)(), std::tuple<ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>&)
clickhouse-server.err.log:2025.09.25 19:42:53.702048 [ 1622 ] {} <Fatal> BaseDaemon: 22. ./src/Common/ThreadPool.h:312: ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&&)::'lambda'()::operator()() @ 0x000000000ff19d26
clickhouse-server.err.log:2025.09.25 19:42:53.708893 [ 1622 ] {} <Fatal> BaseDaemon: 23.0. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ?
clickhouse-server.err.log:2025.09.25 19:42:53.708918 [ 1622 ] {} <Fatal> BaseDaemon: 23.1. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:989: ?
clickhouse-server.err.log:2025.09.25 19:42:53.708927 [ 1622 ] {} <Fatal> BaseDaemon: 23. ./ci/tmp/build/./src/Common/ThreadPool.cpp:812: ThreadPoolImpl<std::thread>::ThreadFromThreadPool::worker() @ 0x000000000ff12020
clickhouse-server.err.log:2025.09.25 19:42:53.722658 [ 1622 ] {} <Fatal> BaseDaemon: 24.0. inlined from ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: decltype(*std::declval<ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>().*std::declval<void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)()>()()) std::__invoke[abi:ne190107]<void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*, void>(void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*&&)
clickhouse-server.err.log:2025.09.25 19:42:53.722691 [ 1622 ] {} <Fatal> BaseDaemon: 24.1. inlined from ./contrib/llvm-project/libcxx/include/__thread/thread.h:192: void std::__thread_execute[abi:ne190107]<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*, 2ul>(std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>&, std::__tuple_indices<2ul>)
clickhouse-server.err.log:2025.09.25 19:42:53.722704 [ 1622 ] {} <Fatal> BaseDaemon: 24. ./contrib/llvm-project/libcxx/include/__thread/thread.h:201: void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x000000000ff17e3b
clickhouse-server.err.log:2025.09.25 19:42:53.722727 [ 1622 ] {} <Fatal> BaseDaemon: 25. ? @ 0x0000000000094ac3
clickhouse-server.err.log:2025.09.25 19:42:53.722739 [ 1622 ] {} <Fatal> BaseDaemon: 26. ? @ 0x0000000000126850
clickhouse-server.err.log:2025.09.25 19:42:53.935549 [ 1622 ] {} <Fatal> BaseDaemon: Integrity check of the executable successfully passed (checksum: 79AB97D48C7413F03C4A3955953DADDC)
clickhouse-server.err.log:2025.09.25 19:42:55.277309 [ 1622 ] {} <Fatal> BaseDaemon: Report this error to https://github.com/ClickHouse/ClickHouse/issues
clickhouse-server.err.log:2025.09.25 19:42:55.277445 [ 1622 ] {} <Fatal> BaseDaemon: Changed settings: min_compress_block_size = 1749775, max_compress_block_size = 377813, max_block_size = 53635, min_external_table_block_size_bytes = 1, max_joined_block_size_rows = 60145, max_insert_threads = 1, max_threads = 3, max_read_buffer_size = 938090, connect_timeout_with_failover_ms = 2000, connect_timeout_with_failover_secure_ms = 3000, use_hedged_requests = false, idle_connection_timeout = 36000, s3_max_get_rps = 1000000, s3_max_get_burst = 2000000, s3_max_put_rps = 1000000, s3_max_put_burst = 2000000, s3_check_objects_after_upload = true, max_remote_read_network_bandwidth = 1000000000000, max_remote_write_network_bandwidth = 1000000000000, max_local_read_bandwidth = 1000000000000, max_local_write_bandwidth = 1000000000000, stream_like_engine_allow_direct_select = true, enable_multiple_prewhere_read_steps = false, replication_wait_for_inactive_replica_timeout = 30, compile_expressions = false, min_count_to_compile_expression = 0, min_count_to_compile_aggregate_expression = 0, group_by_two_level_threshold = 234971, group_by_two_level_threshold_bytes = 8627332, allow_nonconst_timezone_arguments = true, min_chunk_bytes_for_parallel_parsing = 16535326, output_format_parallel_formatting = false, merge_tree_coarse_index_granularity = 7, min_bytes_to_use_direct_io = 10737418240, min_bytes_to_use_mmap_io = 10737418240, log_queries = true, insert_quorum_timeout = 60000, merge_tree_read_split_ranges_into_intersecting_and_non_intersecting_injection_probability = 0., http_wait_end_of_query = true, http_response_buffer_size = 4646614, fsync_metadata = true, query_plan_join_swap_table = false, http_send_timeout = 60., http_receive_timeout = 60., use_index_for_in_with_subqueries_max_values = 1000000000, opentelemetry_start_trace_probability = 0.10000000149011612, enable_vertical_final = false, max_rows_to_read = 20000000, max_bytes_to_read = 1, max_bytes_to_read_leaf = 1000000000000, max_rows_to_group_by = 10000000000, max_bytes_before_externa

Note

02950_part_offset_as_primary_key is in parallel_replicas_blacklist.txt but it's not expected to fail with Logical error: 'Replica 1 decided to read in WithOrder mode, not in Default. This is a bug'.. This issue supposed to be fixed by #80652 but it looks there are still some cases

[zoomxi]

@devcrafter It might be related to #87096. Could you please revert it first? I will resubmit the PR after fixing the issue.

[devcrafter]

@devcrafter It might be related to #87096. Could you please revert it first? I will resubmit the PR after fixing the issue.

@zoomxi Why do you think the change is related? 02950_part_offset_as_primary_key test doesn't deal with projections at all but still fails

[zoomxi]

@devcrafter To be precise, the PR #87096 exposed the issue. I think I've identified the problem which caused by #84971 and will submit a PR to fix it later.