Skip to content

Commit

Permalink
system_keyspace: use system memory for system.raft table
Browse files Browse the repository at this point in the history
`system.raft` was using the "user memory pool", i.e. the
`dirty_memory_manager` for this table was set to
`database::_dirty_memory_manager` (instead of
`database::_system_dirty_memory_manager`).

This meant that if a write workload caused memory pressure on the user
memory pool, internal `system.raft` writes would have to wait for
memtables of user tables to get flushed before the write would proceed.

This was observed in SCT longevity tests which ran a heavy workload on
the cluster and concurrently, schema changes (which underneath use the
`system.raft` table). Raft would often get stuck waiting many seconds
for user memtables to get flushed. More details in issue scylladb#15622.
Experiments showed that moving Raft to system memory fixed this
particular issue, bringing the waits to reasonable levels.

Currently `system.raft` stores only one group, group 0, which is
internally used for cluster metadata operations (schema and topology
changes) -- so it makes sense to keep use system memory.

In the future we'd like to have other groups, for strongly consistent
tables. These groups should use the user memory pool. It means we won't
be able to use `system.raft` for them -- we'll just have to use a
separate table.

Fixes: scylladb#15622
  • Loading branch information
kbr-scylla committed Nov 6, 2023
1 parent 6cc5bca commit 23fea29
Showing 1 changed file with 1 addition and 2 deletions.
3 changes: 1 addition & 2 deletions db/system_keyspace.cc
Original file line number Diff line number Diff line change
Expand Up @@ -1915,8 +1915,7 @@ std::vector<schema_ptr> system_keyspace::all_tables(const db::config& cfg) {

static bool maybe_write_in_user_memory(schema_ptr s) {
return (s.get() == system_keyspace::batchlog().get()) || (s.get() == system_keyspace::paxos().get())
|| s == system_keyspace::v3::scylla_views_builds_in_progress()
|| s == system_keyspace::raft();
|| s == system_keyspace::v3::scylla_views_builds_in_progress();
}

future<> system_keyspace::make(
Expand Down

0 comments on commit 23fea29

Please sign in to comment.