os/bluestore: add multiple finishers to bluestore #10567
Conversation
- The single finisher of a bluestore can be a bottleneck when using an SSD as a backend device. If too much load is given to the single finisher, client-side IO latency increases. So we add multiple finishers to the bluestore, which shows better performance. - ‘bluestore_finisher_threads' option is added to be able to configure the number of finisher threads. Signed-off-by: Ilsoo Byun <ilsoo.byun@sk.com>
| txc->oncommit = NULL; | ||
| } | ||
| while (!txc->oncommits.empty()) { | ||
| finisher.queue(txc->oncommits.front()); | ||
| auto f = txc->oncommits.front(); | ||
| finishers[txc->osr->id % m_finisher_num]->queue(f); |
There was a problem hiding this comment.
I think the way to do this is not to use random ids, but to instead assign a shard id to the sequencer when it is created. This can then map to the same mapping scheme that is distributing pg's across teh op_wq shards in the OSD and the collections across cache shards in BlueStore. Probably we should add the shard_hint to the ObjectStore::Sequencer class and initialize it in the OSD/PG code when the sequencer is created. Elsewhere we use this to do the mapping:
cid.hash_to_shard(num_shards)
So maybe make the config option a bool, and then use the g_conf->osd_op_num_shards config value that also drives op_wq shard and cache shard counts?
There was a problem hiding this comment.
Thanks. I have modified according to your comment.
- a finisher is selected according to the shard id of a sequencer. - the number of finishers is decided by osd_op_num_shards. - 'bluestore_use_multiple_finishers' option is added to be able to configure wheather finsihers is multiple or not. - 'bluetore_finisher_threads' option is removed. Signed-off-by: Ilsoo Byun <ilsoo.byun@sk.com>
|
Does the new scheme still show an improvement for your workload? Do you have any numbers you can share? |
|
Yes, the new scheme has an enhancement of approximately 30%(from 77k iops to 100k iops, 4kb randwrite). But it's not higher than the previous scheme. Both schemes show similar performance. In the previous version, the number of finishers is set to 16, and in the new version, osd_op_num_shards is set to 20, which is the number of finishers. |
| txc->oncommit = NULL; | ||
| } | ||
| while (!txc->oncommits.empty()) { | ||
| finisher.queue(txc->oncommits.front()); | ||
| auto f = txc->oncommits.front(); | ||
| finishers[txc->osr->parent->shard_hint.hash_to_shard(m_finisher_num)]->queue(f); |
There was a problem hiding this comment.
unsigned n = txc->osr->parent->shard_hint.hash_to_shard(m_finisher_num); and then use it for all of these?
|
Great. Do you mind squashing these into one commit? Otherwise, lgtm! |
- The single finisher of a bluestore can be a bottleneck when using an SSD as a backend device. If too much load is given to the single finisher, client-side IO latency increases. So we add multiple finishers to the bluestore, which shows better performance. - 'bluestore_use_multiple_finishers' option is added to be able to configure wheather finsihers is multiple or not. - a finisher is selected according to the shard id of a sequencer. - the number of finishers is decided by osd_op_num_shards. Signed-off-by: Ilsoo Byun <ilsoo.byun@sk.com>
…to multi_finisher
|
@liewegas No. I don't mind. I have squashed them into one commit. Thanks! |
|
#10780 rebased, squash, and config option name changed. |
Signed-off-by: Ilsoo Byun ilsoo.byun@sk.com