Discussion: reworking membership changes #117

ongardie · 2016-04-11T21:58:12Z

This is meant for discussion and is not meant to be merged. See the text in membership.md.

The AppendEntries request handler used to temporarily truncate entries from the store that had indexes overlapping with the request, then it'd go back and add all the entries in the request. This opened a window of vulnerability during which if the server crashed, it could have deleted some log entries that it shouldn't have. For this to happen, the follower would have to receive an AppendEntries request with entries that it already had. In principle, that could be due to a duplicate request (depends on transport behavior) or to a leader with a nextIndex set too low (probably isn't supposed to happen with the back-up-one-entry algorithm). The new code truncates only conflicting entries from the log store, then appends only the new entries. This also fixes a related issue where the last log index and term are set to the last entry in the request, even if the last entry in the request is not the last entry in the log (suppose the request contained an earlier subsequence of the existing log).

This is meant for discussion and is not meant to be merged.

ongardie · 2016-04-11T22:00:51Z

mentioning #84 ("Cleanup Meta Ticket")

slackpad · 2016-04-11T22:34:54Z

membership.md

+
+## Data-centric view
+
+We propose to re-define a *configuration* as a set of servers, where each server includes an address (as it does today) and a mode that is either:


This seems like a good opportunity to give each server something like a GUID that's independent of its address, so that we can eventually support address changes without changing a server's identity.

Want to spec out how that would work? Specifically:

Who would generate the GUID and when?

Where would it be stored?

How would it be reflected in the old AddPeer/RemovePeer or upcoming AddVoter/AddNonvoter/DemoteVoter/RemovePeer API?

I'll spec this out - I think I've got a plan that'll work for this.

slackpad · 2016-04-11T23:23:48Z

@ongardie added some initial thoughts as inline comments. I've got some cycles this week and can work on this with you guys as well if we can figure out some ways to break things up.

ongardie · 2016-04-12T00:33:38Z

@slackpad thanks for the comments. I replied to a few above.

I've got some cycles this week and can work on this with you guys as well if we can figure out some ways to break things up.

As I mentioned, @superfell is out this week. I was going to wait for him to return (he hasn't read this doc, and he might be sitting on some useful changes). We may have to do pair programming or something extreme like that, because I really don't see how to split up the work (pun intended).

If you've got space cycles this week, I think we'd be better off applying them to more parallelizable tasks. For example, how about attacking some more of #84 (AFAIK, multi-row fetching or follower replication are open)?

slackpad · 2016-04-12T02:23:39Z

Yep - I'll take some of those smaller tasks this week while we work through the details here.

slackpad · 2016-04-21T01:52:04Z

membership.md

+We propose the following operations for clients to manipulate the cluster configuration:
+ - AddVoter: server becomes staging unless voter,
+ - AddNonvoter: server becomes nonvoter unless staging or voter,
+ - DemoteVoter: server becomes nonvoter unless absent,


To have more symmetry with AddVoter how about just calling AddNonvoter in order to demote? Either that or we should have a PromoteNonvoter API vs. adding it again.

fd · 2016-06-08T08:34:09Z

Is there a time line for these enhancements?

slackpad · 2016-06-09T02:01:01Z

@fd work is underway now, so hopefully very soon!

…membership Merged in: - issue-84-integration to catch up with the latest changes, - ongardie/truncation (PR hashicorp#113) to avoid a conflict, and - 2c24fd1 (from Simon Fell, splits out configuration change channel) because that's useful here.

This is the main patch for hashicorp#117. It still has several outstanding problems that we'll work through in later commits, but it's good enough to discuss.

ongardie-sfdc · 2016-06-16T23:40:54Z

Added a really, really big diff (687db02) with the bulk of the changes. @superfell gets credit for some of it (especially fixing tests), but I squashed his stuff into the one commit.

@slackpad, @sean-: please review. We should aim to come up with a list of outstanding problems. Some of them are going to be easy, others less so.

slackpad · 2016-06-19T23:33:38Z

@ongardie thanks for this! I'm returning from some travel on Monday so I should be able to make a deep dive here on Tuesday.

slackpad · 2016-06-22T18:01:31Z

raft.go

-		return nil, fmt.Errorf("failed to get list of peers: %v", err)
-	}
-	peers = ExcludePeer(peers, localAddr)
+	localGUID := localAddr // TODO: where should we get this from?


My initial thought was that this would be provided as a new field in conf, and if that's empty then we'd use localAddr as a default and emit a deprecation warning that we'll stop defaulting it at some point in the future.

Ok, that's what I was leaning towards.

sean- · 2016-06-23T05:52:11Z

raft.go

@@ -488,7 +607,7 @@ func (r *Raft) Stats() map[string]string {
 		"fsm_pending":         toString(uint64(len(r.fsmCommitCh))),
 		"last_snapshot_index": toString(lastSnapIndex),
 		"last_snapshot_term":  toString(lastSnapTerm),
-		"num_peers":           toString(uint64(len(r.peers))),
+		"num_servers":         toString(uint64(len(r.configurations.latest.Servers))),


Elsewhere we've taken to including the list of servers along with the number of peers. e.g. https://github.com/hashicorp/nomad/blob/c6b7bb66f309b7fc3ef325b158115aa046dcbf30/nomad/server.go#L905-L906

A list of servers doesn't have all the information in this new world. What is this function used for?

This ends up getting displayed to operators when they run consul info on a server, so the results here can be pretty free-form. This is used for debugging and bug reporting.

A %+v on the configuration might be the best thing here.

This uncovered an issue with: r.leaderState.commitment.match(r.localID, lastIndex) where localAddress was being used incorrectly instead.

ongardie-sfdc · 2016-06-24T22:48:38Z

I believe I've addressed all the actionable code comments above in this latest push (commits 15e3819 to 33b1b0e). I tagged the specific comments that I believe I addressed with the 🎉 reaction (try to avoid using that to indicate celebration).

slackpad · 2016-06-28T02:02:47Z

Went ahead and merged this so we can submit smaller-scoped changes to close this out. Make a list of outstanding items on #84.

ongardie-sfdc · 2016-06-28T02:03:55Z

@slackpad, I wasn't expecting you to merge that! I think there's still some important stuff left to do. For example, we should audit whether this is going to work for rolling upgrades.

ongardie-sfdc · 2016-06-28T02:04:21Z

I think not. For example, new code will create LogConfiguration entries. What will old code do with them?

ongardie-sfdc · 2016-06-28T02:05:50Z

Oh, I see the list on #84 now, fine.

slackpad · 2016-06-28T02:06:08Z

@ongardie I just merged it to the integration branch, not master!

ongardie and others added 3 commits April 2, 2016 13:32

use separate channel for peer changes

2c24fd1

Add membership.md to describe planned changes

b47cfcd

This is meant for discussion and is not meant to be merged.

slackpad reviewed Apr 11, 2016
View reviewed changes

Call r.setLastLog any time entries have been appended

7f565bb

This was referenced Apr 13, 2016

Create an easier way to force step down to a single node cluster hashicorp/consul#1516

Closed

Changing bootstrap_expect at runtime hashicorp/consul#1458

Closed

armon mentioned this pull request Apr 20, 2016

Cleanup Meta Ticket #84

Closed

14 tasks

slackpad reviewed Apr 21, 2016
View reviewed changes

slackpad mentioned this pull request Apr 26, 2016

Server tries to send information to agent although it already left hashicorp/consul#1045

Closed

ongardie added 2 commits May 27, 2016 16:16

Add TODO for appendEntries() tests

ed3396e

Add TODO for broken error case

9c5c0a3

ongardie added 2 commits June 16, 2016 10:10

Rework membership changes

687db02

This is the main patch for hashicorp#117. It still has several outstanding problems that we'll work through in later commits, but it's good enough to discuss.

slackpad reviewed Jun 22, 2016
View reviewed changes

sean- reviewed Jun 23, 2016
View reviewed changes

ongardie added 9 commits June 24, 2016 15:01

Add new types for ServerID and ServerAddress

15e3819

This uncovered an issue with: r.leaderState.commitment.match(r.localID, lastIndex) where localAddress was being used incorrectly instead.

Take ServerID from Config struct when present, warn otherwise

09207cb

Change Stats() to dump latest configuration

671f848

Switch ServerSuffrage enum to iota

4ab6c62

Add missing unit tests for configuration.go

001e8ce

checkConfiguration unit test: check error strings too

5766a7d

Revert accidental change of didWarn := false

91334a2

leaderLoop: use hasVote to avoid looping

625f8dd

Add comment about configurationChangeCh

33b1b0e

slackpad merged commit 0eb014d into hashicorp:issue-84-integration Jun 28, 2016

ongardie-sfdc mentioned this pull request Jul 6, 2016

Split apart and test appendConfigurationEntry #132

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Discussion: reworking membership changes #117

Discussion: reworking membership changes #117

ongardie commented Apr 11, 2016

ongardie commented Apr 11, 2016

slackpad Apr 11, 2016

ongardie Apr 12, 2016

slackpad Apr 12, 2016

slackpad commented Apr 11, 2016

ongardie commented Apr 12, 2016

slackpad commented Apr 12, 2016

slackpad Apr 21, 2016 •

edited

fd commented Jun 8, 2016

slackpad commented Jun 9, 2016

ongardie-sfdc commented Jun 16, 2016

slackpad commented Jun 19, 2016

slackpad Jun 22, 2016

ongardie-sfdc Jun 23, 2016

sean- Jun 23, 2016

ongardie-sfdc Jun 23, 2016

slackpad Jun 23, 2016

slackpad Jun 23, 2016

ongardie-sfdc commented Jun 24, 2016

slackpad commented Jun 28, 2016

ongardie-sfdc commented Jun 28, 2016

ongardie-sfdc commented Jun 28, 2016

ongardie-sfdc commented Jun 28, 2016

slackpad commented Jun 28, 2016


		## Data-centric view

		We propose to re-define a configuration as a set of servers, where each server includes an address (as it does today) and a mode that is either:

Discussion: reworking membership changes #117

Discussion: reworking membership changes #117

Conversation

ongardie commented Apr 11, 2016

ongardie commented Apr 11, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slackpad commented Apr 11, 2016

ongardie commented Apr 12, 2016

slackpad commented Apr 12, 2016

slackpad Apr 21, 2016 • edited

Choose a reason for hiding this comment

fd commented Jun 8, 2016

slackpad commented Jun 9, 2016

ongardie-sfdc commented Jun 16, 2016

slackpad commented Jun 19, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ongardie-sfdc commented Jun 24, 2016

slackpad commented Jun 28, 2016

ongardie-sfdc commented Jun 28, 2016

ongardie-sfdc commented Jun 28, 2016

ongardie-sfdc commented Jun 28, 2016

slackpad commented Jun 28, 2016

slackpad Apr 21, 2016 •

edited