New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

osd: reduce map cache size #15292

Merged
merged 6 commits into from May 28, 2017

Conversation

Projects
None yet
3 participants
@liewegas
Member

liewegas commented May 25, 2017

  • turn off unnecessary diagnostic code that needs old maps (the last big user!)
  • add perfcounters to measure cache performance
  • hint to objectstore to cache osdmap data
  • reduce cache size by about 4x
@jdurgin

looks like the config change test needs updating

liewegas added some commits May 25, 2017

osd: remove osd_enxio_on_misdirected_op option
There is no good reason anyone would want this turned on.

Introduced 923e7f5 (post-kraken), but
backported to kraken and jewel (10.2.6).

Signed-off-by: Sage Weil <sage@redhat.com>
osd: do not bother with misdirected op diagnosis by default
We enable osd_debug_misdirected_ops in QA, but this is wasted effort on
a production cluster.  In particular, it means that a idle client that
sends an op to the wrong OSD based on an old map will require that OSD to
load that old map into memory to decide whether to print a warning... all
on the off-chance that the client is buggy.

Signed-off-by: Sage Weil <sage@redhat.com>
osd: instrument osdmap bl cache hits and misses
Add perfcounters so we can see whether we are missing osdmaps in the
cache.  This will let us tell whether, given a workload or environment,
our osdmap cache might be too small.

Signed-off-by: Sage Weil <sage@redhat.com>
osd: fadvise hint WILL_NEED when reading encoded osdmaps
This way will ensure we cache data for recent osdmaps if we need to for
the benefit of laggy clients... even if (in bluestore's case)
bluestore_default_buffered_reads = false (it's true by default).  This
should mitigate any tail latency/work even if the osdmap cache size is too
small.

Signed-off-by: Sage Weil <sage@redhat.com>
osd: reduce size of osdmap cache, messages
On large clusters, these large caches can be problematic (as maps get big).
We've seen good results with extremely small caches (10s of maps).  Make
a more modest reduction.

Signed-off-by: Sage Weil <sage@redhat.com>
test/osd/osd-config.sh: fix test to isolate cases
The third test (increasing osd_map_max_advance)
was triggering a warning from the 4th case (which
it didn't before).

Signed-off-by: Sage Weil <sage@redhat.com>
@liewegas

This comment has been minimized.

Member

liewegas commented May 26, 2017

i missed an option, and fixed a test.

@tchaikov

This comment has been minimized.

Contributor

tchaikov commented May 26, 2017

/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/osd/osd-config.sh:73: TEST_config_track:  grep 'is not > osd_map_max_advance' td/osd-config/osd.0.log
2017-05-25 21:23:50.905699 7f510fd48c80  0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_map_max_advance (48)
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/osd/osd-config.sh:73: TEST_config_track:  return 1
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/osd/osd-config.sh:34: run:  return 1

i am not sure where osd_map_max_advance = 48 came from. looking..

@tchaikov

ahh, fixed already =)

@liewegas liewegas merged commit d8f3ee1 into ceph:master May 28, 2017

3 checks passed

Signed-off-by all commits in this PR are signed
Details
Unmodifed Submodules submodules for project are unmodified
Details
default Build finished.
Details

@liewegas liewegas deleted the liewegas:wip-map-cache branch May 28, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment