Fix crash when using ReplicatedCollapsingMergeTree with async_insert and async_insert_deduplicate

[al13n321]

Changelog category (leave one):

• Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Fixed a crash when using a combination of ReplicatedCollapsingMergeTree, async_insert, and async_insert_deduplicate. On older versions, use optimize_on_insert=0 as a workaround.

Basically optimize_on_insert and async_insert_deduplicate are incompatible. This PR makes the latter implicitly disable the former, in a not particularly elegant way.

optimize_on_insert code was mutating a block, then async_insert_deduplicate code expected the block to not have been mutated (for filtering subranges of rows within the block that correspond to original INSERT queries).

Part of the stack trace (to make this searchable):

2023.07.01 04:13:48.823371 [ 3087524 ] {} <Fatal> BaseDaemon: 3.1. inlined from ./build/./base/base/../base/unaligned.h:49: unsigned long unalignedLoadEndian<(std::endian)57005, unsigned long>(void const*)
2023.07.01 04:13:48.823419 [ 3087524 ] {} <Fatal> BaseDaemon: 3.2. inlined from ./build/./base/base/../base/unaligned.h:70: unsigned long unalignedLoadLittleEndian<unsigned long>(void const*)
2023.07.01 04:13:48.823448 [ 3087524 ] {} <Fatal> BaseDaemon: 3. ./build/./src/Columns/ColumnVector.cpp:75: DB::ColumnVector<long>::updateHashWithValue(unsigned long, SipHash&) const @ 0x0000000017ae3ff7 in /home/ubuntu/clickhouse-private/build/programs/clickhouse
2023.07.01 04:13:49.329465 [ 3087524 ] {} <Fatal> BaseDaemon: 4.1. inlined from ./build/./contrib/llvm-project/libcxx/include/__iterator/wrap_iter.h:100: std::__wrap_iter<COW<DB::IColumn>::immutable_ptr<DB::IColumn>*>::operator++[abi:v15000]()
2023.07.01 04:13:49.329513 [ 3087524 ] {} <Fatal> BaseDaemon: 4.2. inlined from ./build/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:238: DB::(anonymous namespace)::getHashesForBlocks(DB::BlockWithPartition&, String)
2023.07.01 04:13:49.329521 [ 3087524 ] {} <Fatal> BaseDaemon: 4. ./build/./src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp:449: DB::ReplicatedMergeTreeSinkImpl<true>::consume(DB::Chunk) @ 0x000000001849ce72 in /home/ubuntu/clickhouse-private/build/programs/clickhouse
2023.07.01 04:13:49.350623 [ 3087524 ] {} <Fatal> BaseDaemon: 5.1. inlined from ./build/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: ~shared_ptr
2023.07.01 04:13:49.350669 [ 3087524 ] {} <Fatal> BaseDaemon: 5.2. inlined from ./build/./src/Processors/Chunk.h:32: ~Chunk
2023.07.01 04:13:49.350678 [ 3087524 ] {} <Fatal> BaseDaemon: 5. ./build/./src/Processors/Sinks/SinkToStorage.cpp:18: DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001897067b in /home/ubuntu/clickhouse-private/build/programs/clickhouse
2023.07.01 04:13:49.434222 [ 3087524 ] {} <Fatal> BaseDaemon: 6.1. inlined from ./build/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: ~shared_ptr
2023.07.01 04:13:49.434262 [ 3087524 ] {} <Fatal> BaseDaemon: 6.2. inlined from ./build/./src/Processors/Chunk.h:32: ~Chunk
2023.07.01 04:13:49.434272 [ 3087524 ] {} <Fatal> BaseDaemon: 6.3. inlined from ./build/./src/Processors/Transforms/ExceptionKeepingTransform.cpp:151: operator()
2023.07.01 04:13:49.434294 [ 3087524 ] {} <Fatal> BaseDaemon: 6.4. inlined from ./build/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval<DB::ExceptionKeepingTransform::work()::$_1&>()()) std::__invoke[abi:v15000]<DB::ExceptionKeepingTransform::work()::$_1&>(DB::ExceptionKeepingTransform::work()::$_1&)
2023.07.01 04:13:49.434311 [ 3087524 ] {} <Fatal> BaseDaemon: 6.5. inlined from ./build/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479: void std::__invoke_void_return_wrapper<void, true>::__call<DB::ExceptionKeepingTransform::work()::$_1&>(DB::ExceptionKeepingTransform::work()::$_1&)
2023.07.01 04:13:49.434322 [ 3087524 ] {} <Fatal> BaseDaemon: 6.6. inlined from ./build/./contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func<DB::ExceptionKeepingTransform::work()::$_1, void ()>::operator()[abi:v15000]()
2023.07.01 04:13:49.434331 [ 3087524 ] {} <Fatal> BaseDaemon: 6. ./build/./contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::ExceptionKeepingTransform::work()::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x00000000188c6a6b in /home/ubuntu/clickhouse-private/build/programs/clickhouse                                                                                                                                                                                                              
2023.07.01 04:13:49.456492 [ 3087524 ] {} <Fatal> BaseDaemon: 7. ./build/./src/Processors/Transforms/ExceptionKeepingTransform.cpp:103: DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x00000000188c68f0 in /home/ubuntu/clickhouse-private/build/programs/clickhouse
2023.07.01 04:13:49.470104 [ 3087524 ] {} <Fatal> BaseDaemon: 8.1. inlined from ./build/./contrib/llvm-project/libcxx/include/__functional/function.h:818: ~__policy_func
2023.07.01 04:13:49.470142 [ 3087524 ] {} <Fatal> BaseDaemon: 8.2. inlined from ./build/./contrib/llvm-project/libcxx/include/__functional/function.h:1174: ~function
2023.07.01 04:13:49.470150 [ 3087524 ] {} <Fatal> BaseDaemon: 8. ./build/./src/Processors/Transforms/ExceptionKeepingTransform.cpp:151: DB::ExceptionKeepingTransform::work() @ 0x00000000188c63b0 in /home/ubuntu/clickhouse-private/build/programs/clickhouse
2023.07.01 04:13:49.499401 [ 3087524 ] {} <Fatal> BaseDaemon: 9.1. inlined from ./build/./src/Processors/Executors/ExecutionThreadContext.cpp:0: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*)
2023.07.01 04:13:49.499440 [ 3087524 ] {} <Fatal> BaseDaemon: 9. ./build/./src/Processors/Executors/ExecutionThreadContext.cpp:92: DB::ExecutionThreadContext::executeTask() @ 0x00000000186e14f8 in /home/ubuntu/clickhouse-private/build/programs/clickhouse
2023.07.01 04:13:49.552160 [ 3087524 ] {} <Fatal> BaseDaemon: 10. ./build/./src/Processors/Executors/PipelineExecutor.cpp:255: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x00000000186d9110 in /home/ubuntu/clickhouse-private/build/programs/clickhouse
2023.07.01 04:13:49.571620 [ 3087524 ] {} <Fatal> BaseDaemon: 11.1. inlined from ./build/./src/Processors/Executors/PipelineExecutor.cpp:221: ~shared_ptr
2023.07.01 04:13:49.571659 [ 3087524 ] {} <Fatal> BaseDaemon: 11. ./build/./src/Processors/Executors/PipelineExecutor.cpp:380: DB::PipelineExecutor::executeImpl(unsigned long) @ 0x00000000186d885c in /home/ubuntu/clickhouse-private/build/programs/clickhouse                                                      
2023.07.01 04:13:49.593984 [ 3087524 ] {} <Fatal> BaseDaemon: 12.1. inlined from ./build/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: std::unique_ptr<DB::ExecutingGraph, std::default_delete<DB::ExecutingGraph>>::operator->[abi:v15000]() const
2023.07.01 04:13:49.594027 [ 3087524 ] {} <Fatal> BaseDaemon: 12. ./build/./src/Processors/Executors/PipelineExecutor.cpp:116: DB::PipelineExecutor::execute(unsigned long) @ 0x00000000186d8637 in /home/ubuntu/clickhouse-private/build/programs/clickhouse
2023.07.01 04:13:49.606755 [ 3087524 ] {} <Fatal> BaseDaemon: 13. ./build/./src/Processors/Executors/CompletedPipelineExecutor.cpp:0: DB::CompletedPipelineExecutor::execute() @ 0x00000000186d7697 in /home/ubuntu/clickhouse-private/build/programs/clickhouse                                                       
2023.07.01 04:13:49.688715 [ 3087524 ] {} <Fatal> BaseDaemon: 14. ./build/./src/Interpreters/AsynchronousInsertQueue.cpp:532: DB::AsynchronousInsertQueue::processData(DB::AsynchronousInsertQueue::InsertQuery, std::unique_ptr<DB::AsynchronousInsertQueue::InsertData, std::default_delete<DB::AsynchronousInsertQueue::InsertData>>, std::shared_ptr<DB::Context const>) @ 0x0000000016bdb42d in /home/ubuntu/clickhouse-private/build/programs/clickhouse
2023.07.01 04:13:49.772499 [ 3087524 ] {} <Fatal> BaseDaemon: 15.1. inlined from ./build/./src/Core/Settings.h:1037: ~Settings
2023.07.01 04:13:49.772537 [ 3087524 ] {} <Fatal> BaseDaemon: 15.2. inlined from ./build/./src/Interpreters/AsynchronousInsertQueue.h:46: ~InsertQuery
2023.07.01 04:13:49.772548 [ 3087524 ] {} <Fatal> BaseDaemon: 15.3. inlined from ./build/./src/Interpreters/AsynchronousInsertQueue.cpp:195: operator()
2023.07.01 04:13:49.772575 [ 3087524 ] {} <Fatal> BaseDaemon: 15.4. inlined from ./build/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394: decltype(std::declval<DB::AsynchronousInsertQueue::scheduleDataProcessingJob(DB::AsynchronousInsertQueue::InsertQuery const&, std::unique_ptr<DB::AsynchronousInsertQueue::InsertData, std::default_delete<DB::AsynchronousInsertQueue::InsertData>>, std::shared_ptr<DB::Context const>)::$_0&>()()) std::__invoke[abi:v15000]<DB::AsynchronousInsertQueue::scheduleDataProcessingJob(DB::AsynchronousInsertQueue::InsertQuery const&, std::unique_ptr<DB::AsynchronousInsertQueue::InsertData, std::default_delete<DB::AsynchronousInsertQueue::InsertData>>, std::shared_ptr<DB::Context const>)::$_0&>(DB::AsynchronousInsertQueue::scheduleDataProcessingJob(DB::AsynchronousInsertQueue::InsertQuery const&, std::unique_ptr<DB::AsynchronousInsertQueue::InsertData, std::default_delete<DB::AsynchronousInsertQueue::InsertData>>, std::shared_ptr<DB::Context const>)::$_0&)
2023.07.01 04:13:49.772597 [ 3087524 ] {} <Fatal> BaseDaemon: 15.5. inlined from ./build/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479: void std::__invoke_void_return_wrapper<void, true>::__call<DB::AsynchronousInsertQueue::scheduleDataProcessingJob(DB::AsynchronousInsertQueue::InsertQuery const&, std::unique_ptr<DB::AsynchronousInsertQueue::InsertData, std::default_delete<DB::AsynchronousInsertQueue::InsertData>>, std::shared_ptr<DB::Context const>)::$_0&>(DB::AsynchronousInsertQueue::scheduleDataProcessingJob(DB::AsynchronousInsertQueue::InsertQuery const&, std::unique_ptr<DB::AsynchronousInsertQueue::InsertData, std::default_delete<DB::AsynchronousInsertQueue::InsertData>>, std::shared_ptr<DB::Context const>)::$_0&)
2023.07.01 04:13:49.772612 [ 3087524 ] {} <Fatal> BaseDaemon: 15.6. inlined from ./build/./contrib/llvm-project/libcxx/include/__functional/function.h:235: std::__function::__default_alloc_func<DB::AsynchronousInsertQueue::scheduleDataProcessingJob(DB::AsynchronousInsertQueue::InsertQuery const&, std::unique_ptr<DB::AsynchronousInsertQueue::InsertData, std::default_delete<DB::AsynchronousInsertQueue::InsertData>>, std::shared_ptr<DB::Context const>)::$_0, void ()>::operator()[abi:v15000]()
2023.07.01 04:13:49.772632 [ 3087524 ] {} <Fatal> BaseDaemon: 15. ./build/./contrib/llvm-project/libcxx/include/__functional/function.h:716: void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::AsynchronousInsertQueue::scheduleDataProcessingJob(DB::AsynchronousInsertQueue::InsertQuery const&, std::unique_ptr<DB::AsynchronousInsertQueue::InsertData, std::default_delete<DB::AsynchronousInsertQueue::InsertData>>, std::shared_ptr<DB::Context const>)::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x0000000016bdc92c in /home/ubuntu/clickhouse-private/build/programs/clickhouse
2023.07.01 04:13:49.796957 [ 3087524 ] {} <Fatal> BaseDaemon: 16.1. inlined from ./build/./base/base/../base/wide_integer_impl.h:796: bool wide::integer<128ul, unsigned int>::_impl::operator_eq<wide::integer<128ul, unsigned int>>(wide::integer<128ul, unsigned int> const&, wide::integer<128ul, unsigned int> const&)

(For a quick manual repro, copy-paste this megabyte of text into clickhouse client -nm: https://pastila.nl/?1aa66bad/d77374fa44563912bdde9fc10c1076d0 )

[robot-clickhouse-ci-1]

This is an automated comment for commit 7d4ab52 with description of existing statuses. It's updated for the latest CI running
The full report is available here
The overall status of the commit is 🔴 failure

Check name	Description	Status
CI running	A meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR	🟡 pending
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	🟢 success
Mergeable Check	Checks if all other necessary checks are successful	🔴 failure
Push to Dockerhub	The check for building and pushing the CI related docker images to docker hub	🟢 success
Style Check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	🔴 failure

[antonio2368]

Duplicate of #51676