Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

ring mgr crash creates confused cluster #166

Closed
rzezeski opened this Issue · 1 comment

3 participants

@rzezeski

NOTE: This should be an extremely rare incident and is easily fixed
via a riak-admin call or a node restart (see end of issue for
work-around).

If, for whatever reason, the ring mgr crashes it results in the node
thinking it is a cluster of it's own. However, it will still have all
it's distributed Erlang connections in place and the other nodes will
consider it as part of the cluster.

To be preemptive, I don't think allowing unexpected msgs is the fix
(like we do with so many other servers). In fact, I think an unknown
msg should still crash the mgr.

After talking with @jtuple it sounds like the fix has to do with
changing the mgr to be the process responsible for reading the ring.

member_status before confused cluster

$ dev1/bin/riak-admin member_status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid      25.0%      --      'dev1@127.0.0.1'
valid      25.0%      --      'dev2@127.0.0.1'
valid      25.0%      --      'dev3@127.0.0.1'
valid      25.0%      --      'dev4@127.0.0.1'
-------------------------------------------------------------------------------
Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0

$ dev2/bin/riak-admin member_status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid      25.0%      --      'dev1@127.0.0.1'
valid      25.0%      --      'dev2@127.0.0.1'
valid      25.0%      --      'dev3@127.0.0.1'
valid      25.0%      --      'dev4@127.0.0.1'
-------------------------------------------------------------------------------
Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0

now crash the ring mgr

dev1/bin/riak attach
> gen_server:call(riak_core_ring_manager, fooey).
... crash report ...

check member_status again

dev1/bin/riak-admin member_status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid     100.0%      --      'dev1@127.0.0.1'
-------------------------------------------------------------------------------
Valid:1 / Leaving:0 / Exiting:0 / Joining:0 / Down:0

dev2/bin/riak-admin member_status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid      25.0%      --      'dev1@127.0.0.1'
valid      25.0%      --      'dev2@127.0.0.1'
valid      25.0%      --      'dev3@127.0.0.1'
valid      25.0%      --      'dev4@127.0.0.1'
-------------------------------------------------------------------------------
Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0

investigate ring

> {ok, RawRing} = riak_core_ring_manager:get_raw_ring().
> sets:to_list(sets:from_list([Owner || {_,Owner} <- riak_core_ring:all_owners(RawRing)])).
['dev1@127.0.0.1']

work-around

The work-around is to join the node with the crashed ring mgr to any
other node in the cluster. It's important that it the join is
initiated by the node with the crashed mgr because joins can only be
performed by nodes not already part of a cluster (and in this case,
the in-memory parts of Riak think the node is not a member of the
cluster).

Also, if the node restarts it will read the correct ring off disk and
you don't have to do anything special.

dev/dev1/bin/riak-admin join 'dev2@127.0.0.1'
Sent join request to dev2@127.0.0.1

dev/dev1/bin/riak-admin member_status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid      25.0%      --      'dev1@127.0.0.1'
valid      25.0%      --      'dev2@127.0.0.1'
valid      25.0%      --      'dev3@127.0.0.1'
valid      25.0%      --      'dev4@127.0.0.1'
-------------------------------------------------------------------------------

dev2/bin/riak-admin member_status
================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid      25.0%      --      'dev1@127.0.0.1'
valid      25.0%      --      'dev2@127.0.0.1'
valid      25.0%      --      'dev3@127.0.0.1'
valid      25.0%      --      'dev4@127.0.0.1'
-------------------------------------------------------------------------------
Valid:4 / Leaving:0 / Exiting:0 / Joining:0 / Down:0
@jonmeredith
Owner

Probably an artifact of the start up sequence. It would be better for the ring manager to load the ring file directly, rather than have it passed in. It's no longer necessary with the change that starts all the fallbacks over time #163

@jtuple jtuple closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.