jewel: systemd/ceph-disk: reduce ceph-disk flock contention #12210
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
"ceph-disk trigger" invocation is currently performed in a mutually exclusive fashion, with each call first taking an flock on the path /var/lock/ceph-disk. On systems with a lot of osds, this leads to a large amount of lock contention during boot-up, and can cause some service instances to trip the 120 second timeout. Take an flock on a device specific path instead of /var/lock/ceph-disk, so that concurrent "ceph-disk trigger" invocations are permitted for independent osds. This greatly reduces lock contention and consequently the chance of service timeout. Per-device concurrency restrictions required for http://tracker.ceph.com/issues/13160 are maintained. Fixes: http://tracker.ceph.com/issues/18060 Signed-off-by: David Disseldorp <ddiss@suse.de> (cherry picked from commit 8a62cbc)
ddiss
changed the title
ghost
self-assigned this
tchaikov
changed the title
ghost
pushed a commit
that referenced
this pull request
Dec 5, 2016
… flock contention Reviewed-by: Loic Dachary <ldachary@redhat.com>
ghost
changed the base branch from
|
Ping - anything I can do to help move this along? |
|
@dachary I guess all this needs is a ceph-disk suite? I'll schedule one. |
|
This PR was included in this [1] integration branch which already passed a ceph-disk suite at [2]. [1] http://tracker.ceph.com/issues/17851#note-17 |
|
there is a systemd test in smoke which can use used as well, FYI |
|
@vasukulkarni Thanks for the heads-up, but I couldn't find it in the |
This pull request was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
"ceph-disk trigger" invocation is currently performed in a mutually
exclusive fashion, with each call first taking an flock on the path
/var/lock/ceph-disk. On systems with a lot of osds, this leads to a
large amount of lock contention during boot-up, and can cause some
service instances to trip the 120 second timeout.
Take an flock on a device specific path instead of /var/lock/ceph-disk,
so that concurrent "ceph-disk trigger" invocations are permitted for
independent osds. This greatly reduces lock contention and consequently
the chance of service timeout. Per-device concurrency restrictions
required for http://tracker.ceph.com/issues/13160 are maintained.
Fixes: http://tracker.ceph.com/issues/18060
Signed-off-by: David Disseldorp ddiss@suse.de
(cherry picked from commit 8a62cbc)