leveldb: increase leveldb defaults (filestore, keyvaluestore)

[liewegas]

The FileStore's leveldb currently uses libleveldb's defaults for cache and
write buffer size, which are both 4 MB. Increase the cache size to 128MB and
the write buffer to 8MB.

Signed-off-by: Sage Weil sage@inktank.com

[gregsfortytwo]

Why? Doing so is a pretty dramatic increase in our per-OSD steady-state RAM usage and I haven't heard any evidence that we need these values to be larger.

[liewegas]

Because the default is tiny, and leveldb is hit pretty hard for rgw workloads. And possibly others in general due to the xattr spillover stuff. 128mb seems like a reasonable increase to me, but 32 or 64 is still a big step up.

[gregsfortytwo]

Sure it's hit hard for those workloads, but is it hit hard in a way that caching and buffering usefully improves? We bumped up the monitor defaults because we had compelling evidence that it provided a smoother user experience (things were breaking without it), but if we've got similar evidence for the OSDs I haven't heard anything about it.

[onlyjob]

I'd like to vote for this change.
I backported it (as below) and applied to 0.72.2. After upgrading OSDs I observe nice performance boost to CephFS -- less latency, and faster response especially on random access. Although I did no benchmarks I can feel the difference on PC with /home on CephFS.

Thank you.

--- a/src/common/config_opts.h
+++ b/src/common/config_opts.h
@@ -486,10 +486,10 @@
 OPTION(osd_op_history_duration, OPT_U32, 600) // Oldest completed op to track
 OPTION(osd_target_transaction_size, OPT_INT, 30)     // to adjust various transactions that batch smaller items
 OPTION(osd_failsafe_full_ratio, OPT_FLOAT, .97) // what % full makes an OSD "full" (failsafe)
 OPTION(osd_failsafe_nearfull_ratio, OPT_FLOAT, .90) // what % full makes an OSD near full (failsafe)
-OPTION(osd_leveldb_write_buffer_size, OPT_U64, 0) // OSD's leveldb write buffer size
-OPTION(osd_leveldb_cache_size, OPT_U64, 0) // OSD's leveldb cache size
+OPTION(osd_leveldb_write_buffer_size, OPT_U64, 8*1024*1024) // OSD's leveldb write buffer size
+OPTION(osd_leveldb_cache_size, OPT_U64, 256*1024*1024) // OSD's leveldb cache size
 OPTION(osd_leveldb_block_size, OPT_U64, 0) // OSD's leveldb block size
 OPTION(osd_leveldb_bloom_size, OPT_INT, 0) // OSD's leveldb bloom bits per entry
 OPTION(osd_leveldb_max_open_files, OPT_INT, 0) // OSD's leveldb max open files
 OPTION(osd_leveldb_compression, OPT_BOOL, true) // OSD's leveldb uses compression