New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#17980: MDS client blacklisting and blacklist on eviction #14610

Merged
merged 20 commits into from May 23, 2017

Conversation

Projects
None yet
3 participants
@jcsp
Contributor

jcsp commented Apr 18, 2017

This is the big rework/strengthening of eviction:

  • Evicting a client from one MDS daemon evicts it from all
  • Once a client is evicted it cannot reconnect to any MDS daemon
  • Evicted clients are blacklisted in the OSDMap, so they can no longer write data to OSDs
  • Clients notice their own eviction and do some teardown so that they can error out any operations in flight and be unmountable.
if (g_conf->mds_session_blacklist_on_timeout) {
std::stringstream ss;
mds->kill_session(session->info.inst.name.num(), false, true,

This comment has been minimized.

@jcsp

jcsp Apr 18, 2017

Contributor

I should rename this!

@@ -1666,6 +1667,11 @@ int Client::make_request(MetaRequest *request,
if (request->aborted())
break;
if (blacklisted) {
request->abort(-EIO);

This comment has been minimized.

@jcsp

jcsp Apr 18, 2017

Contributor

Let's replace this with ESHUTDOWN

@batrick

This comment has been minimized.

Member

batrick commented May 9, 2017

@jcsp, FYI this has conflicts now.

std::set<entity_addr_t> delta_set;
std::set_difference(
new_set.begin(), old_set.begin(), new_set.end(), old_set.end(),
std::inserter(delta_set, delta_set.begin()));

This comment has been minimized.

@ukernel

ukernel May 19, 2017

Member

see http://en.cppreference.com/w/cpp/algorithm/set_difference. this should be "new_set.begin(), new_set.end(), old_set.begin(), old_set.end()"

}
};
auto background_blacklist = [this, session_id, kill_mds_session, cmd](std::function<void ()> fn){

This comment has been minimized.

@ukernel

ukernel May 19, 2017

Member

kill_mds_session is not required here

@ukernel

This comment has been minimized.

Member

ukernel commented May 19, 2017

LGTM

jcsp added some commits Feb 16, 2017

osdc: expose new blacklist entries from Objecter
For the MDS which needs to know about new blacklist entries
in order to evict corresponding MDS sessions.

Fixes: http://tracker.ceph.com/issues/17980
Signed-off-by: John Spray <john.spray@redhat.com>
osd/OSDMap: std::set version of get_blacklist
Signed-off-by: John Spray <john.spray@redhat.com>
mds: apply OSDMap blacklist to CephFS clients
Signed-off-by: John Spray <john.spray@redhat.com>
mds: remove stale comment
The code the comment is talking about went
away in 2490a0a.

Signed-off-by: John Spray <john.spray@redhat.com>
mon: permit MDS daemons to do blacklisting
Signed-off-by: John Spray <john.spray@redhat.com>
mds: blacklist clients on eviction
Fixes: http://tracker.ceph.com/issues/9754
Signed-off-by: John Spray <john.spray@redhat.com>
mds: update osdmap epoch barrier on eviction
Signed-off-by: John Spray <john.spray@redhat.com>
mds: update osd epoch barrier when handling blacklist
Signed-off-by: John Spray <john.spray@redhat.com>
client: detect blacklisting of myself
EIO all operations when this happens, and do a cut-down
unmount() so that once the user has noticed his mount
is broken, he can unmount it without getting stuck
waiting for flushes/caps/sessions.

Signed-off-by: John Spray <john.spray@redhat.com>
client: fix unmounting while blacklisted
Previously, if you ever made it as far as ~Client
with any inodes in your cache, you'd hit the ObjectCacher
is_locked_by_me assertion during release_set on those inode's osets.

Signed-off-by: John Spray <john.spray@redhat.com>
mds: make blacklisting configurable in kill_session
This makes the code a bit awkward because we now have
4x paths through this function (blocking, noblocking, blacklisting,
nonblacklisting).

Signed-off-by: John Spray <john.spray@redhat.com>
mds: hook up autoclose to blacklisting eviction
Signed-off-by: John Spray <john.spray@redhat.com>
qa: add test_filelock_eviction
To check that eviction is releasing flocks.

Signed-off-by: John Spray <john.spray@redhat.com>
mds: tidy up 'kill_session' naming
We had two kill_sessions.  Leave Server::kill_session (the
one that actually operates on MDS sessions only), and rename
the MDSRank one to evict_client, because it does more than
just kill the MDS session.

Signed-off-by: John Spray <john.spray@redhat.com>
mds: improve eviction usability and update docs
The "session ls" and "session evict" are now
"client ls" and "client evict" (the old ones
are still there for backwards compatibility).

The automatic client eviction now emits
cluster logs that call the client by its
friendly name (usually the hostname).

Signed-off-by: John Spray <john.spray@redhat.com>
qa: log whitelist for eviction messages
Signed-off-by: John Spray <john.spray@redhat.com>
qa: remove outdated TODO in TestVolumeClient
Signed-off-by: John Spray <john.spray@redhat.com>
qa: clean up whitespace in test_misc.py
Signed-off-by: John Spray <john.spray@redhat.com>
qa: update TestVolumeClient for new blacklisting
Blacklisted clients will now proactively fail
outstanding operations, rather than blocking.

Signed-off-by: John Spray <john.spray@redhat.com>
qa: refine assert_session_count (don't count killing)
Signed-off-by: John Spray <john.spray@redhat.com>

@jcsp jcsp merged commit 8d85c52 into ceph:master May 23, 2017

3 checks passed

Signed-off-by all commits in this PR are signed
Details
Unmodifed Submodules submodules for project are unmodified
Details
default Build finished.
Details

@jcsp jcsp deleted the jcsp:wip-17980 branch May 23, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment