Logical error: 'Cannot find transaction (115, 33, 5a0ac955-d43c-4a5d-8392-a3aca7bdba75) that has started mutation mutation_12.txt that is going to be applied to part all_7_7_0'

[kssenii]

https://s3.amazonaws.com/clickhouse-test-reports/51622/55769cd9d586488f3e257fcd0c857a29dfdf2fad/stress_test__asan_/fatal_messages.txt

/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 12:54:36.137085 [ 2159 ] {62fc8802-63ed-4e06-aeec-ceb83728efc8::all_9_9_0_12} <Fatal> : Logical error: 'Unexpected CSN state: 116'.
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:10:42.971445 [ 1903 ] {} <Fatal> Application: Child process was terminated by signal 6.
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.456306 [ 22877 ] {} <Fatal> : Logical error: 'Cannot find transaction (115, 33, 5a0ac955-d43c-4a5d-8392-a3aca7bdba75) that has started mutation mutation_12.txt that is going to be applied to part all_7_7_0'.
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.457612 [ 23201 ] {} <Fatal> BaseDaemon: ########## Short fault info ############
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.457709 [ 23201 ] {} <Fatal> BaseDaemon: (version 23.8.1.1, build id: C4288DEA6B23070DF84DD4EA969A5319172EABCB, git hash: 2573461b05e2b94de4aead6ac61f55a661722c8b) (from thread 22877) Received signal 6
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.457799 [ 23201 ] {} <Fatal> BaseDaemon: Signal description: Aborted
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.457859 [ 23201 ] {} <Fatal> BaseDaemon: 
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.457931 [ 23201 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007f32f40f7a7c 0x00007f32f40a3476 0x00007f32f40897f3 0x00005617762cd448 0x00005617762ce486 0x000056178d1e3d1b 0x000056178d1a5495 0x000056178d1a7f40 0x000056178c53f45e 0x0000561787041698 0x0000561787048b44 0x000056178704a0e2 0x00005617765a67f2 0x00005617765b4acd 0x00007f32f40f5b43 0x00007f32f4187a00
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.458159 [ 23201 ] {} <Fatal> BaseDaemon: ########################################
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.458376 [ 23201 ] {} <Fatal> BaseDaemon: (version 23.8.1.1, build id: C4288DEA6B23070DF84DD4EA969A5319172EABCB, git hash: 2573461b05e2b94de4aead6ac61f55a661722c8b) (from thread 22877) (no query) Received signal Aborted (6)
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.458434 [ 23201 ] {} <Fatal> BaseDaemon: 
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.458588 [ 23201 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007f32f40f7a7c 0x00007f32f40a3476 0x00007f32f40897f3 0x00005617762cd448 0x00005617762ce486 0x000056178d1e3d1b 0x000056178d1a5495 0x000056178d1a7f40 0x000056178c53f45e 0x0000561787041698 0x0000561787048b44 0x000056178704a0e2 0x00005617765a67f2 0x00005617765b4acd 0x00007f32f40f5b43 0x00007f32f4187a00
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.458667 [ 23201 ] {} <Fatal> BaseDaemon: 3. ? @ 0x00007f32f40f7a7c in ?
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.458724 [ 23201 ] {} <Fatal> BaseDaemon: 4. ? @ 0x00007f32f40a3476 in ?
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.458789 [ 23201 ] {} <Fatal> BaseDaemon: 5. ? @ 0x00007f32f40897f3 in ?
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.648384 [ 23106 ] {} <Fatal> : Logical error: 'Data part all_9_9_0 is Outdated and has creation TID (113, 34, 5a0ac955-d43c-4a5d-8392-a3aca7bdba75) and CSN 115, but does not have removal tid. It's a bug or a result of manual intervention.'.
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.680517 [ 23203 ] {} <Fatal> BaseDaemon: ########## Short fault info ############
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.689004 [ 23203 ] {} <Fatal> BaseDaemon: (version 23.8.1.1, build id: C4288DEA6B23070DF84DD4EA969A5319172EABCB, git hash: 2573461b05e2b94de4aead6ac61f55a661722c8b) (from thread 23106) Received signal 6
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.700429 [ 23203 ] {} <Fatal> BaseDaemon: Signal description: Aborted
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.703955 [ 23203 ] {} <Fatal> BaseDaemon: 
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.707027 [ 23203 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007f32f40f7a7c 0x00007f32f40a3476 0x00007f32f40897f3 0x00005617762cd448 0x00005617762ce486 0x000056178c97a31e 0x000056178c7ef918 0x000056178c9629a0 0x0000561786f50aad 0x0000561785fcec1e 0x00005617765afcfc 0x00005617765ba160 0x00005617765a67f2 0x00005617765b4acd 0x00007f32f40f5b43 0x00007f32f4187a00
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.708003 [ 23203 ] {} <Fatal> BaseDaemon: ########################################
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.711460 [ 23203 ] {} <Fatal> BaseDaemon: (version 23.8.1.1, build id: C4288DEA6B23070DF84DD4EA969A5319172EABCB, git hash: 2573461b05e2b94de4aead6ac61f55a661722c8b) (from thread 23106) (no query) Received signal Aborted (6)
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.712531 [ 23203 ] {} <Fatal> BaseDaemon: 
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.714864 [ 23203 ] {} <Fatal> BaseDaemon: Stack trace: 0x00007f32f40f7a7c 0x00007f32f40a3476 0x00007f32f40897f3 0x00005617762cd448 0x00005617762ce486 0x000056178c97a31e 0x000056178c7ef918 0x000056178c9629a0 0x0000561786f50aad 0x0000561785fcec1e 0x00005617765afcfc 0x00005617765ba160 0x00005617765a67f2 0x00005617765b4acd 0x00007f32f40f5b43 0x00007f32f4187a00
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.715766 [ 23203 ] {} <Fatal> BaseDaemon: 3. ? @ 0x00007f32f40f7a7c in ?
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.717842 [ 23203 ] {} <Fatal> BaseDaemon: 4. ? @ 0x00007f32f40a3476 in ?
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.719907 [ 23203 ] {} <Fatal> BaseDaemon: 5. ? @ 0x00007f32f40897f3 in ?
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:40.787588 [ 23201 ] {} <Fatal> BaseDaemon: 6. ./build_docker/./src/Common/Exception.cpp:43: DB::abortOnFailedAssertion(String const&) @ 0x000000001d4d4448 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:41.025534 [ 23203 ] {} <Fatal> BaseDaemon: 6. ./build_docker/./src/Common/Exception.cpp:43: DB::abortOnFailedAssertion(String const&) @ 0x000000001d4d4448 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:41.119688 [ 23201 ] {} <Fatal> BaseDaemon: 7.1. inlined from ./build_docker/./src/Common/Exception.cpp:68: DB::handle_error_code(String const&, int, bool, std::vector<void*, std::allocator<void*>> const&)
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:15:41.129292 [ 23201 ] {} <Fatal> BaseDaemon: 7. ./build_docker/./src/Common/Exception.cpp:102: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d4d5486 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:36.713130 [ 23203 ] {} <Fatal> BaseDaemon: 7.1. inlined from ./build_docker/./src/Common/Exception.cpp:68: DB::handle_error_code(String const&, int, bool, std::vector<void*, std::allocator<void*>> const&)
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:36.713599 [ 23203 ] {} <Fatal> BaseDaemon: 7. ./build_docker/./src/Common/Exception.cpp:102: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001d4d5486 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:38.032137 [ 23201 ] {} <Fatal> BaseDaemon: 8.1. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: String::__is_long[abi:v15000]() const
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:38.032192 [ 23201 ] {} <Fatal> BaseDaemon: 8.2. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/string:2333: ~basic_string
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:38.032233 [ 23201 ] {} <Fatal> BaseDaemon: 8.3. inlined from ./build_docker/./src/Common/Exception.h:61: ~MessageMasked
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:38.032274 [ 23201 ] {} <Fatal> BaseDaemon: 8.4. inlined from ./build_docker/./src/Common/Exception.h:73: Exception
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:38.032303 [ 23201 ] {} <Fatal> BaseDaemon: 8. ./build_docker/./src/Common/Exception.h:101: DB::Exception::Exception<DB::TransactionID&, String&, String const&>(int, FormatStringHelperImpl<std::type_identity<DB::TransactionID&>::type, std::type_identity<String&>::type, std::type_identity<String const&>::type>, DB::TransactionID&, String&, String const&) @ 0x00000000343ead1b in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:38.747935 [ 23203 ] {} <Fatal> BaseDaemon: 8.1. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: String::__is_long[abi:v15000]() const
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:38.748018 [ 23203 ] {} <Fatal> BaseDaemon: 8.2. inlined from ./build_docker/./contrib/llvm-project/libcxx/include/string:2333: ~basic_string
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:38.748069 [ 23203 ] {} <Fatal> BaseDaemon: 8.3. inlined from ./build_docker/./src/Common/Exception.h:61: ~MessageMasked
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:38.748135 [ 23203 ] {} <Fatal> BaseDaemon: 8.4. inlined from ./build_docker/./src/Common/Exception.h:73: Exception
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:38.748175 [ 23203 ] {} <Fatal> BaseDaemon: 8. ./build_docker/./src/Common/Exception.h:101: DB::Exception::Exception<String const&, DB::TransactionID&, unsigned long&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type, std::type_identity<DB::TransactionID&>::type, std::type_identity<unsigned long&>::type>, String const&, DB::TransactionID&, unsigned long&) @ 0x0000000033b8131e in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:38.786210 [ 23201 ] {} <Fatal> BaseDaemon: 9. ./build_docker/./src/Storages/StorageMergeTree.cpp:1173: DB::StorageMergeTree::selectPartsToMutate(std::shared_ptr<DB::StorageInMemoryMetadata const> const&, String&, std::shared_ptr<DB::RWLockImpl::LockHolderImpl>&, std::unique_lock<std::mutex>&) @ 0x00000000343ac495 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:39.406301 [ 23201 ] {} <Fatal> BaseDaemon: 10. ./build_docker/./src/Storages/StorageMergeTree.cpp:0: DB::StorageMergeTree::scheduleDataProcessingJob(DB::BackgroundJobsAssignee&) @ 0x00000000343aef40 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:39.461073 [ 23201 ] {} <Fatal> BaseDaemon: 11. ./build_docker/./src/Storages/MergeTree/BackgroundJobsAssignee.cpp:129: DB::BackgroundJobsAssignee::threadFunc() @ 0x000000003374645e in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:39.570834 [ 23201 ] {} <Fatal> BaseDaemon: 12. ./build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:0: DB::BackgroundSchedulePoolTaskInfo::execute() @ 0x000000002e248698 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:39.664857 [ 23201 ] {} <Fatal> BaseDaemon: 13. ./build_docker/./src/Core/BackgroundSchedulePool.cpp:0: DB::BackgroundSchedulePool::threadFunction() @ 0x000000002e24fb44 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.err.log:2023.08.20 13:16:39.746085 [ 23201 ] {} <Fatal> BaseDaemon: 14.1. inlined from ./build_docker/./src/Common/ThreadPool.h:243: operator()

[qoega]

https://s3.amazonaws.com/clickhouse-test-reports/0/f76c9e10ff5450d4aa392d94aac24a9ad334b54a/stress_test__tsan_.html