flush all InMemoryDataParts when wal is not enabled

[nautaa]

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Flush all In-Memory data parts when WAL is not enabled while shutdown server or detaching table.

close #17733

[nautaa]

@Mergifyio update

[mergify]

update

✅ Branch has been successfully updated

[qoega]

Can you elaborate how this tests helps to check that changes work or not?

[nautaa]

Can you elaborate how this tests helps to check that changes work or not?

Before this change, in memory parts with disabled wal are not flush to disk after detach table or shutdown server.
So the test result select count(*) from mem_part_flush will be inconsistent after detach and attach table.

[tavplubix]

https://s3.amazonaws.com/clickhouse-test-reports/0/d1d872329c570b432b50c780517a8065dabfe793/stress_test__debug__actions_.html

/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.723509 [ 516 ] {} <Fatal> BaseDaemon: (version 21.13.1.889, build id: 6A196D5789C8B62F) (from thread 515) Terminate called for uncaught exception:
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.723584 [ 516 ] {} <Fatal> BaseDaemon: Code: 84. DB::Exception: Could not flush part '/var/lib/clickhouse/store/ba3/ba35f7bd-335c-489c-ba35-f7bd335ca89c/all_1_1_0/'. Part in /var/lib/clickhouse/store/ba3/ba35f7bd-335c-489c-ba35-f7bd335ca89c/all_1_1_0 already exists. (DIRECTORY_ALREADY_EXISTS), Stack trace (when copying this message, always include the lines below):
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.723648 [ 516 ] {} <Fatal> BaseDaemon: 
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.723702 [ 516 ] {} <Fatal> BaseDaemon: 0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:133: std::exception::capture() @ 0x15974782 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.723757 [ 516 ] {} <Fatal> BaseDaemon: 1. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:111: std::exception::exception() @ 0x15974753 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.723816 [ 516 ] {} <Fatal> BaseDaemon: 2. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x284561e0 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.723886 [ 516 ] {} <Fatal> BaseDaemon: 3. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:55: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x1595620e in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.723946 [ 516 ] {} <Fatal> BaseDaemon: 4. ./obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeDataPartInMemory.cpp:86: DB::MergeTreeDataPartInMemory::flushToDisk(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&) const @ 0x238a4623 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.724010 [ 516 ] {} <Fatal> BaseDaemon: 5. ./obj-x86_64-linux-gnu/../src/Storages/MergeTree/MergeTreeData.cpp:1540: DB::MergeTreeData::flushAllInMemoryPartsIfNeeded() @ 0x2380e1ab in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.724073 [ 516 ] {} <Fatal> BaseDaemon: 6. ./obj-x86_64-linux-gnu/../src/Storages/StorageMergeTree.cpp:145: DB::StorageMergeTree::flush() @ 0x23b1fd55 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.724127 [ 516 ] {} <Fatal> BaseDaemon: 7. ./obj-x86_64-linux-gnu/../src/Databases/DatabasesCommon.cpp:258: DB::DatabaseWithOwnTablesBase::shutdown() @ 0x221dc287 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.724181 [ 516 ] {} <Fatal> BaseDaemon: 8. ./obj-x86_64-linux-gnu/../src/Databases/DatabaseReplicated.cpp:759: DB::DatabaseReplicated::shutdown() @ 0x221caf3a in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.724234 [ 516 ] {} <Fatal> BaseDaemon: 9. ./obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:183: DB::DatabaseCatalog::shutdownImpl() @ 0x22422674 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.724287 [ 516 ] {} <Fatal> BaseDaemon: 10. ./obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:593: DB::DatabaseCatalog::shutdown() @ 0x224271b7 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.724345 [ 516 ] {} <Fatal> BaseDaemon: 11. ./obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:330: DB::ContextSharedPart::shutdown() @ 0x22392a26 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.724403 [ 516 ] {} <Fatal> BaseDaemon: 12. ./obj-x86_64-linux-gnu/../src/Interpreters/Context.cpp:2641: DB::Context::shutdown() @ 0x22386499 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.724457 [ 516 ] {} <Fatal> BaseDaemon: 13. ./obj-x86_64-linux-gnu/../programs/server/Server.cpp:1073: DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_6::operator()() const @ 0x15a3f942 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.724560 [ 516 ] {} <Fatal> BaseDaemon: 14. ./obj-x86_64-linux-gnu/../base/base/../base/scope_guard.h:95: basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_6>::invoke() @ 0x15a3f4dd in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.724621 [ 516 ] {} <Fatal> BaseDaemon: 15. ./obj-x86_64-linux-gnu/../base/base/../base/scope_guard.h:44: basic_scope_guard<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&)::$_6>::~basic_scope_guard() @ 0x15a3a455 in /usr/bin/clickhouse
/var/log/clickhouse-server/clickhouse-server.log.2:2021.12.19 11:07:40.724683 [ 516 ] {} <Fatal> BaseDaemon: 16. ./obj-x86_64-linux-gnu/../programs/server/Server.cpp:1616: DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0x15a37ed