internal_keyspace extention: enhance the semantics also to flushes

commit 7c8c020 introduced a new type of a keyspace, an internal keyspace
It defined the semantics for this internal keyspace, this keyspace is
somewhat a hybrid between system and user keyspace.

Here we extend the semantics to include also flushes, meaning that
flushes will be done using the system dirty_mamory_manager. This is
in order to allow inter dependencies between internal tables and user
tables and prevent deadlocks.

One example of such a deadlock is our `replicated_key_provider`
encryption on the enterprise version. The deadlock occur because in some
circumstances, an encrypted user table flush is dependant upon the
`encrypted_keys` table being flushed but since the requests are
serialized, we get a deadlock.

Tests: unit tests dev + debug
The deadlock dtest reproducer:
encryption_at_rest_test.py::TestEncryptionAtRest::test_reboot

Fixes #14529

Signed-off-by: Eliran Sinvani <eliransin@scylladb.com>

Closes #14547

replica/database.cc

@@ -1455,6 +1455,10 @@ future<> database::create_in_memory_keyspace(const lw_shared_ptr<keyspace_metada
         // don't make system keyspace writes wait for user writes (if under pressure)
         kscfg.dirty_memory_manager = &_system_dirty_memory_manager;
     }
+    if (extensions().is_extension_internal_keyspace(ksm->name())) {
+        // don't make internal keyspaces write wait for user writes (if under pressure), and also to avoid possible deadlocks.
+        kscfg.dirty_memory_manager = &_system_dirty_memory_manager;
+    }
     keyspace ks(ksm, std::move(kscfg), erm_factory);
     co_await ks.create_replication_strategy(get_shared_token_metadata(), ksm->strategy_options());
     _keyspaces.emplace(ksm->name(), std::move(ks));