New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os/bluestore: cap rocksdb cache size #15786

Merged
merged 5 commits into from Jul 7, 2017

Conversation

Projects
None yet
3 participants
@markhpc
Member

markhpc commented Jun 20, 2017

This PR introduces a new bluestore tunable to limit the amount of cache assigned to rocksdb. This, combined with a high bluestore_cache_kv_ratio, allows us to favor rocksdb's cache at very low cache values but favor bluestore's onode cache at high cache values. This improves 4k random write performance when cache values are low.

  • 1 NVMe OSD
  • 1 512GB RBD Image
  • 64 PGs
  • 4K random writes
  • 128 iodepth

Filestore:
Baseline: 16.1K IOPS

Bluestore Master:
0.25GB Cache: 8.8K IOPS
0.50GB Cache: 10K IOPS
1.00GB Cache: 11.9K IOPS
2.00GB Cache: 17.2K IOPS
3.00GB Cache: 30.9K IOPS
4.00GB Cache: 30.5K IOPS

Bluestore wip-bluestore-cache-behavior:
0.25GB Cache: 12.4K IOPS
0.50GB Cache: 13.4K IOPS
1.00GB Cache: 13.2K IOPS
2.00GB Cache: 16.9K IOPS
3.00GB Cache: 30.9K IOPS
4.00GB Cache: 30.8K IOPS

@liewegas liewegas changed the title from Wip bluestore cache behavior to os/bluestore: cap rocksdb cache size Jun 20, 2017

@markhpc

This comment has been minimized.

Show comment
Hide comment
@markhpc

markhpc Jun 27, 2017

Member

Ok, cleaned this up based on Igor's comments and squashed things down since it was a bit verbose.

Member

markhpc commented Jun 27, 2017

Ok, cleaned this up based on Igor's comments and squashed things down since it was a bit verbose.

@ifed01

ifed01 approved these changes Jun 27, 2017

@liewegas

This comment has been minimized.

Show comment
Hide comment
@liewegas

liewegas Jun 30, 2017

Member

@markhpc mind rebasing? otherwise looks good to merge.

Member

liewegas commented Jun 30, 2017

@markhpc mind rebasing? otherwise looks good to merge.

markhpc added some commits Jun 20, 2017

kv/RocksDBStore: default to disable row cache.
Signed-off-by: Mark Nelson <mnelson@redhat.com>
common/config_opts: Better rocksdb cache size description.
Signed-off-by: Mark Nelson <mnelson@redhat.com>
os/bluestore: limit kv cache size.
Signed-off-by: Mark Nelson <mnelson@redhat.com>
kv/RocksDBStore: explicitly disable block_cache when set to 0.
Signed-off-by: Mark Nelson <mnelson@redhat.com>

@liewegas liewegas added the needs-qa label Jul 6, 2017

@liewegas liewegas added this to the luminous milestone Jul 6, 2017

common/config_opts: favor bluestore kv cache before meta until max.
Signed-off-by: Mark Nelson <mnelson@redhat.com>

@liewegas liewegas merged commit 145999a into ceph:master Jul 7, 2017

3 of 4 checks passed

make check (arm64) make check failed
Details
Signed-off-by all commits in this PR are signed
Details
Unmodified Submodules submodules for project are unmodified
Details
make check make check succeeded
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment