os/bluestore: [RFC] merge multiple TransContext into a batch to improve performance

[ifed01]

This is somewhat hackery approach to merge multiple TransContexts into a single batch to submit them in a single step. The code relies on RocksDB WriteBatch internals to merge multiple batches into a single one. Unfortunately more elegant solution to collect operations out of RocksDB WriteBatch and then build it from scratch isn't beneficial from performance POV as we overburden kv_sync_thread. (See code at (https://github.com/ifed01/ceph/tree/wip-bluestore-large-batch-prod)
This PR is actually a question if we want to proceed this way as well as a summary on obtained numbers.
Results below are for FIO bluestore plugin for both highly concurrent 4K appends and overwrites using 2 NVME drives. (min_alloc_size=4K too). Each case was run for 3 times

Original code (kv_finisher tread PR + intrusive lists for TransContextx)
Append:
Run 1 = 370 Mb/s , run 2 = 367 Mb/s, run3 = 358 Mb/s
Overwrite:
Run 1 = 329 Mb/s , run 2 = 333 Mb/s, run3 = 329 Mb/s

This PR code on top of above-mentioned original one
Append:
Run 1 = 419 Mb/s , run 2 = 413 Mb/s, run3 = 420 Mb/s
Overwrite:
Run 1 = 338 Mb/s , run 2 = 338 Mb/s, run3 = 345 Mb/s

Failed approach to collect KV operations out of RocksDB and build a resulting batch before the submit
Append:
Run 1 = 368 Mb/s , run 2 = 375 Mb/s, run3 = 373 Mb/s
Overwrite:
Run 1 = 270 Mb/s , run 2 = 272 Mb/s, run3 = 278 Mb/s

[liewegas]

This looks pretty straightforward! I think if we go down this path, though, we'd want to patch upstream rocksdb properly to implement the transaction append. And make txc0 reserve enough memory based on what we expect to see for the whole list, etc.

Alternatively, rocksdb could provide a submit_transaction call that takes a vector of writebatches and does the same thing internally without the memcpy. (That would probably be more efficient, fwiw.)

or, we could batch explicitly in bluestore. (Maybe this is what adam's appraoch did? I don't remember)

• attach the transaction to the OpSequencer
• give it a mutex and bool sealed; flag
• when we create a TransContext, use the osr's txc if it is not sealed. otherwise, grab a new one.
• hold hte mutex over prepare_transaction
• in kv sync thread, claim ownership of the transaction, seal it, submit it, etc.

The downside to that approach is that prepare_transaction might be slow (esp if there is a read/modify/write) and that may block up kv sync thread. I don't think there is a way around that without doing a memcpy like your approach. Which probably means that's the way to go...