New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

luminous: core: os/bluestore: cache autotuning and memory limit #24065

Merged
merged 6 commits into from Sep 24, 2018

Conversation

Projects
None yet
6 participants
@jdurgin
Member

jdurgin commented Sep 12, 2018

http://tracker.ceph.com/issues/37341


This is a backport of:

#22009
#22838
#22970

These make bluestore much simpler to configure, since users no longer need to worry about bluestore's cache size or internal details of what is allocated where.

Instead, the osd_memory_target tunable is all that is needed. Since bluestore cannot use the kernel's page cache, the default osd memory target is 4GB. This has no effect on filestore.

markhpc added some commits Apr 26, 2018

common/PriorityCache: First pass at prototype.
Signed-off-by: Mark Nelson <mnelson@redhat.com>
(cherry picked from commit cb46541)

 Conflicts:
	src/CMakeLists.txt
	src/common/legacy_config_opts.h
	src/common/options.cc
	src/kv/KeyValueDB.h
	src/kv/RocksDBStore.cc
	src/kv/RocksDBStore.h
	src/os/bluestore/BlueStore.cc

conflicts due to rocksdb iterator method name changes and renamed
config option (bluestore_cache_kv_max) - resolution trivial.
os/bluestore: Fix potential uint64_t to int conversion issue.
Signed-off-by: Mark Nelson <mnelson@redhat.com>
(cherry picked from commit 1d51e76)
src/kv: Initial import of a custom RocksDB cache.
Signed-off-by: Mark Nelson <mnelson@redhat.com>
(cherry picked from commit 9dd6b28)

Conflicts:
	src/kv/RocksDBStore.cc
trivial - g_conf has no operator()
kv/rocksdb_cache/BinnedLRUCache: Don't promote data to the high pri p…
…ool.

Signed-off-by: Mark Nelson <mnelson@redhat.com>
(cherry picked from commit 269713d)
kv/RocksDBStore: Handle nullptr if clock cache is chosen.
Signed-off-by: Mark Nelson <mnelson@redhat.com>
(cherry picked from commit 9202285)

Conflicts:
	src/kv/RocksDBStore.cc

trivial - g_conf operator()
os/bluestore: limit OSD memory usage by tuning the cache size.
Signed-off-by: Mark Nelson <mnelson@redhat.com>
(cherry picked from commit 1b8a87b)

Conflicts:
	src/common/options.cc
	src/os/bluestore/BlueStore.cc
trivial - no g_conf operator. or option TYPE_SIZE

@jdurgin jdurgin added the bluestore label Sep 12, 2018

@jdurgin jdurgin requested a review from markhpc Sep 12, 2018

@jdurgin

This comment has been minimized.

@liewegas liewegas changed the title from bluestore: cache autotuning and memory limit to luminous: os/bluestore: cache autotuning and memory limit Sep 12, 2018

@liewegas liewegas added this to the luminous milestone Sep 12, 2018

@jdurgin jdurgin requested a review from liewegas Sep 12, 2018

@yuriw

This comment has been minimized.

Contributor

yuriw commented Sep 15, 2018

@yuriw

This comment has been minimized.

Contributor

yuriw commented Sep 20, 2018

@yuriw

This comment has been minimized.

Contributor

yuriw commented Sep 24, 2018

@liewegas @markhpc Pls approve, it passed tests

@liewegas

let's not forget we need this backported to mimic, too, so that users don't see a regression when they upgrade! @jdurgin

@yuriw yuriw added the backport label Sep 24, 2018

@yuriw

This comment has been minimized.

Contributor

yuriw commented Sep 24, 2018

@smithfarm @theanalyst FYI backport

@yuriw yuriw merged commit 8b61511 into ceph:luminous Sep 24, 2018

4 checks passed

Docs: build check OK - docs built
Details
Signed-off-by all commits in this PR are signed
Details
Unmodified Submodules submodules for project are unmodified
Details
make check make check succeeded
Details

@smithfarm smithfarm changed the title from luminous: os/bluestore: cache autotuning and memory limit to luminous: core: os/bluestore: cache autotuning and memory limit Oct 26, 2018

@jdurgin jdurgin deleted the jdurgin:wip-luminous-cache-autotune branch Nov 6, 2018

@smithfarm

This comment has been minimized.

Contributor

smithfarm commented Dec 12, 2018

The mimic backport @liewegas asked for was done by @jdurgin in #25283

Thanks, @jdurgin

@smithfarm smithfarm removed the backport label Dec 12, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment