TiFlash crash for cdc compatibility

[lilinghai]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1. create cluster with upstream tidb and downstream tidb
2. using cdc sync data between upstream and downstream
3. create a table in upstream tidb
4. alter the table set tiflash replica 1 in downstream tidb

2. What did you expect to see? (Required)

3. What did you see instead (Required)

TiFlash crash and can't restart

[2023/04/03 08:14:41.818 +00:00] [ERROR] [Exception.cpp:90] ["Code: 49, e.displayText() = DB::Exception: invalid flag 83 in write cf: physical_table_id=88: (while preHandleSnapshot region_id=113, index=7, term=6), e.what() = DB::Exception, Stack trace:\n\n\n       0x1b6379e\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+28719006]\n                \tdbms/src/Common/Exception.h:46\n       0x7ef6764\tDB::RegionCFDataBase<DB::RegionWriteCFDataTrait>::insert(DB::StringObject<true>&&, DB::StringObject<false>&&) [tiflash+133130084]\n                \tdbms/src/Storages/Transaction/RegionCFDataBase.cpp:46\n       0x7e6b3cb\tDB::DM::SSTFilesToBlockInputStream::read() [tiflash+132559819]\n                \tdbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp:139\n       0x732315b\tDB::DM::readNextBlock(std::__1::shared_ptr<DB::IBlockInputStream> const&) [tiflash+120729947]\n                \tdbms/src/Storages/DeltaMerge/DeltaMergeHelpers.h:258\n       0x7e6db52\tDB::DM::PKSquashingBlockInputStream<true>::read() [tiflash+132569938]\n                \tdbms/src/Storages/DeltaMerge/PKSquashingBlockInputStream.h:68\n       0x732315b\tDB::DM::readNextBlock(std::__1::shared_ptr<DB::IBlockInputStream> const&) [tiflash+120729947]\n                \tdbms/src/Storages/DeltaMerge/DeltaMergeHelpers.h:258\n       0x1a658f5\tDB::DM::DMVersionFilterBlockInputStream<1>::initNextBlock() [tiflash+27678965]\n                \tdbms/src/Storages/DeltaMerge/DMVersionFilterBlockInputStream.h:139\n       0x1a633d5\tDB::DM::DMVersionFilterBlockInputStream<1>::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool) [tiflash+27669461]\n                \tdbms/src/Storages/DeltaMerge/DMVersionFilterBlockInputStream.cpp:51\n       0x7e6d258\tDB::DM::BoundedSSTFilesToBlockInputStream::read() [tiflash+132567640]\n                \tdbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp:309\n       0x1b1fe20\tDB::DM::SSTFilesToDTFilesOutputStream<std::__1::shared_ptr<DB::DM::BoundedSSTFilesToBlockInputStream> >::write() [tiflash+28442144]\n                \tdbms/src/Storages/DeltaMerge/SSTFilesToDTFilesOutputStream.cpp:228\n       0x7e60f1f\tDB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+132517663]\n                \tdbms/src/Storages/Transaction/ApplySnapshot.cpp:365\n       0x7e605e4\tDB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::TMTContext&) [tiflash+132515300]\n                \tdbms/src/Storages/Transaction/ApplySnapshot.cpp:280\n       0x7ed31a5\tPreHandleSnapshot [tiflash+132985253]\n                \tdbms/src/Storages/Transaction/ProxyFFI.cpp:626\n  0x7fbb921e3c0c\tproxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h04c2a27e9ef15cbf [libtiflash_proxy.so+23309324]\n  0x7fbb91d44149\tengine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::h9ee55cdd322f8c62 [libtiflash_proxy.so+18461001]\n  0x7fbb9286b166\tyatp::task::future::RawTask$LT$F$GT$::poll::h4a3e5758388fdb81 [libtiflash_proxy.so+30155110]\n  0x7fbb941d627b\t_$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::ha71ef630d60068b8 [libtiflash_proxy.so+56808059]\n  0x7fbb941c6ccc\tstd::sys_common::backtrace::__rust_begin_short_backtrace::hc212b72837caebfb [libtiflash_proxy.so+56745164]\n  0x7fbb941c774c\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::he019a920f95839e2 [libtiflash_proxy.so+56747852]\n  0x7fbb93911b35\tstd::sys::unix::thread::Thread::new::thread_start::hd2791a9cabec1fda [libtiflash_proxy.so+47614773]\n                \t/rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys/unix/thread.rs:108\n  0x7fbb90878802\tstart_thread [libc.so.6+653314]\n  0x7fbb90818314\t__GI___clone [libc.so.6+258836]"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=15]
[2023/04/03 08:15:01.635 +00:00] [ERROR] [Exception.cpp:90] ["Code: 49, e.displayText() = DB::Exception: invalid flag 83 in write cf: physical_table_id=88: (while preHandleSnapshot region_id=113, index=7, term=6), e.what() = DB::Exception, Stack trace:\n\n\n       0x1b6379e\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+28719006]\n                \tdbms/src/Common/Exception.h:46\n       0x7ef6764\tDB::RegionCFDataBase<DB::RegionWriteCFDataTrait>::insert(DB::StringObject<true>&&, DB::StringObject<false>&&) [tiflash+133130084]\n                \tdbms/src/Storages/Transaction/RegionCFDataBase.cpp:46\n       0x7e6b3cb\tDB::DM::SSTFilesToBlockInputStream::read() [tiflash+132559819]\n                \tdbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp:139\n       0x732315b\tDB::DM::readNextBlock(std::__1::shared_ptr<DB::IBlockInputStream> const&) [tiflash+120729947]\n                \tdbms/src/Storages/DeltaMerge/DeltaMergeHelpers.h:258\n       0x7e6db52\tDB::DM::PKSquashingBlockInputStream<true>::read() [tiflash+132569938]\n                \tdbms/src/Storages/DeltaMerge/PKSquashingBlockInputStream.h:68\n       0x732315b\tDB::DM::readNextBlock(std::__1::shared_ptr<DB::IBlockInputStream> const&) [tiflash+120729947]\n                \tdbms/src/Storages/DeltaMerge/DeltaMergeHelpers.h:258\n       0x1a658f5\tDB::DM::DMVersionFilterBlockInputStream<1>::initNextBlock() [tiflash+27678965]\n                \tdbms/src/Storages/DeltaMerge/DMVersionFilterBlockInputStream.h:139\n       0x1a633d5\tDB::DM::DMVersionFilterBlockInputStream<1>::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool) [tiflash+27669461]\n                \tdbms/src/Storages/DeltaMerge/DMVersionFilterBlockInputStream.cpp:51\n       0x7e6d258\tDB::DM::BoundedSSTFilesToBlockInputStream::read() [tiflash+132567640]\n                \tdbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp:309\n       0x1b1fe20\tDB::DM::SSTFilesToDTFilesOutputStream<std::__1::shared_ptr<DB::DM::BoundedSSTFilesToBlockInputStream> >::write() [tiflash+28442144]\n                \tdbms/src/Storages/DeltaMerge/SSTFilesToDTFilesOutputStream.cpp:228\n       0x7e60f1f\tDB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+132517663]\n                \tdbms/src/Storages/Transaction/ApplySnapshot.cpp:365\n       0x7e605e4\tDB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::TMTContext&) [tiflash+132515300]\n                \tdbms/src/Storages/Transaction/ApplySnapshot.cpp:280\n       0x7ed31a5\tPreHandleSnapshot [tiflash+132985253]\n                \tdbms/src/Storages/Transaction/ProxyFFI.cpp:626\n  0x7f35e14ffc0c\tproxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h04c2a27e9ef15cbf [libtiflash_proxy.so+23309324]\n  0x7f35e1060149\tengine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::h9ee55cdd322f8c62 [libtiflash_proxy.so+18461001]\n  0x7f35e1b87166\tyatp::task::future::RawTask$LT$F$GT$::poll::h4a3e5758388fdb81 [libtiflash_proxy.so+30155110]\n  0x7f35e34f227b\t_$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::ha71ef630d60068b8 [libtiflash_proxy.so+56808059]\n  0x7f35e34e2ccc\tstd::sys_common::backtrace::__rust_begin_short_backtrace::hc212b72837caebfb [libtiflash_proxy.so+56745164]\n  0x7f35e34e374c\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::he019a920f95839e2 [libtiflash_proxy.so+56747852]\n  0x7f35e2c2db35\tstd::sys::unix::thread::Thread::new::thread_start::hd2791a9cabec1fda [libtiflash_proxy.so+47614773]\n                \t/rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys/unix/thread.rs:108\n  0x7f35dfb94802\tstart_thread [libc.so.6+653314]\n  0x7f35dfb34314\t__GI___clone [libc.so.6+258836]"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=15]
[2023/04/03 08:15:35.266 +00:00] [ERROR] [Exception.cpp:90] ["Code: 49, e.displayText() = DB::Exception: invalid flag 83 in write cf: physical_table_id=88: (while preHandleSnapshot region_id=113, index=7, term=6), e.what() = DB::Exception, Stack trace:\n\n\n       0x1b6379e\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+28719006]\n                \tdbms/src/Common/Exception.h:46\n       0x7ef6764\tDB::RegionCFDataBase<DB::RegionWriteCFDataTrait>::insert(DB::StringObject<true>&&, DB::StringObject<false>&&) [tiflash+133130084]\n                \tdbms/src/Storages/Transaction/RegionCFDataBase.cpp:46\n       0x7e6b3cb\tDB::DM::SSTFilesToBlockInputStream::read() [tiflash+132559819]\n                \tdbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp:139\n       0x732315b\tDB::DM::readNextBlock(std::__1::shared_ptr<DB::IBlockInputStream> const&) [tiflash+120729947]\n                \tdbms/src/Storages/DeltaMerge/DeltaMergeHelpers.h:258\n       0x7e6db52\tDB::DM::PKSquashingBlockInputStream<true>::read() [tiflash+132569938]\n                \tdbms/src/Storages/DeltaMerge/PKSquashingBlockInputStream.h:68\n       0x732315b\tDB::DM::readNextBlock(std::__1::shared_ptr<DB::IBlockInputStream> const&) [tiflash+120729947]\n                \tdbms/src/Storages/DeltaMerge/DeltaMergeHelpers.h:258\n       0x1a658f5\tDB::DM::DMVersionFilterBlockInputStream<1>::initNextBlock() [tiflash+27678965]\n                \tdbms/src/Storages/DeltaMerge/DMVersionFilterBlockInputStream.h:139\n       0x1a633d5\tDB::DM::DMVersionFilterBlockInputStream<1>::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool) [tiflash+27669461]\n                \tdbms/src/Storages/DeltaMerge/DMVersionFilterBlockInputStream.cpp:51\n       0x7e6d258\tDB::DM::BoundedSSTFilesToBlockInputStream::read() [tiflash+132567640]\n                \tdbms/src/Storages/DeltaMerge/SSTFilesToBlockInputStream.cpp:309\n       0x1b1fe20\tDB::DM::SSTFilesToDTFilesOutputStream<std::__1::shared_ptr<DB::DM::BoundedSSTFilesToBlockInputStream> >::write() [tiflash+28442144]\n                \tdbms/src/Storages/DeltaMerge/SSTFilesToDTFilesOutputStream.cpp:228\n       0x7e60f1f\tDB::KVStore::preHandleSSTsToDTFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::DM::FileConvertJobType, DB::TMTContext&) [tiflash+132517663]\n                \tdbms/src/Storages/Transaction/ApplySnapshot.cpp:365\n       0x7e605e4\tDB::KVStore::preHandleSnapshotToFiles(std::__1::shared_ptr<DB::Region>, DB::SSTViewVec, unsigned long, unsigned long, DB::TMTContext&) [tiflash+132515300]\n                \tdbms/src/Storages/Transaction/ApplySnapshot.cpp:280\n       0x7ed31a5\tPreHandleSnapshot [tiflash+132985253]\n                \tdbms/src/Storages/Transaction/ProxyFFI.cpp:626\n  0x7f040b280c0c\tproxy_ffi::engine_store_helper_impls::_$LT$impl$u20$proxy_ffi..interfaces..root..DB..EngineStoreServerHelper$GT$::pre_handle_snapshot::h04c2a27e9ef15cbf [libtiflash_proxy.so+23309324]\n  0x7f040ade1149\tengine_store_ffi::core::forward_raft::snapshot::pre_handle_snapshot_impl::h9ee55cdd322f8c62 [libtiflash_proxy.so+18461001]\n  0x7f040b908166\tyatp::task::future::RawTask$LT$F$GT$::poll::h4a3e5758388fdb81 [libtiflash_proxy.so+30155110]\n  0x7f040d27327b\t_$LT$yatp..task..future..Runner$u20$as$u20$yatp..pool..runner..Runner$GT$::handle::ha71ef630d60068b8 [libtiflash_proxy.so+56808059]\n  0x7f040d263ccc\tstd::sys_common::backtrace::__rust_begin_short_backtrace::hc212b72837caebfb [libtiflash_proxy.so+56745164]\n  0x7f040d26474c\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::he019a920f95839e2 [libtiflash_proxy.so+56747852]\n  0x7f040c9aeb35\tstd::sys::unix::thread::Thread::new::thread_start::hd2791a9cabec1fda [libtiflash_proxy.so+47614773]\n                \t/rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys/unix/thread.rs:108\n  0x7f0409915802\tstart_thread [libc.so.6+653314]\n  0x7f04098b5314\t__GI___clone [libc.so.6+258836]"] [source="DB::RawCppPtr DB::PreHandleSnapshot(DB::EngineStoreServerWrap *, DB::BaseBuffView, uint64_t, DB::SSTViewVec, uint64_t, uint64_t)"] [thread_id=15]

4. What is your TiFlash version? (Required)

master