replace goraft

[bmizerany]

We - the etcd team - are working on a replacing goraft with a new raft implementation intended to live in the etcd codebase under etcd/raft. Our reasons for this are as follows:

After discussions with users and maintainers of the goraft library we decided that there were some core things that could only be achieved by starting fresh with a new API and tighter codebase:

• Simplicity - Currently the implementation we use has a large surface area and tangles "core raft" with other essential pieces like transports, storage, and timing.

Our proposed implementation abstracts the store, timing, and network away and leaves us with with a simple state-machine that can be used by other packages that implement their own transport and storage. Implementing a transport is an easier task when it isn't intertwined with the Raft state-machine.

The proposed implementation takes a new approach. It assumes messages can come out of order, so transports need not rely on careful bookkeeping. The new etcd/raft also avoids having a concept of a transport - it simply queues outgoing messages, and committable entires for the transport and storage systems to dequeue and save/send when ready.

• Testability - Proving the stability of any codebase is hard. But, it is easier with smaller interfaces between parts of the code.

By removing the dependency of wall-clock time we can write isolated, wait-free, and race-free unit tests that run in an amount of time measured in milliseconds.

Removing the knowledge of an explicit network allows us to - amongst other things - emulate network partitions and perform other types of fuzzing to prove correctness right in our unit tests as opposed to larger systems that require complicated test harnesess and iptables manipulation. That isn't to say the latter method isn't necessary for a complete picture, but being able to emulate most of the chaos you encounter in a real network in our unit tests allows us to catch regressions much, much faster.

Since Raft is so essential to etcd we felt that investing time and energy into making a better library with these two goals in mind is worth the effort. Please review the PR if you have the chance. We intend for this package to be useful for other projects, too. Also, if you have the need for a simple, pure Go Raft package in your application, please take a look.

I also want to give a huge thank you to Ben and Xiang who put in a ton of work to make github.com/goraft/raft happen. It has gotten etcd a long way. That said they both feel having a few implementations of such an important building block is OK and that new cleaner APIs are possible.

[bmhatfield]

I think it would be awesome if this RAFT implementation was go-gettable (ie; it's own github repo). Thanks for this work!

[bmizerany]

@bmhatfield It will be go-gettable. Once merged, you can: $ go get github.com/coreos/etcd/raft.

[peterbourgon]

Out of curiosity, can you summarize how this will be different than hashicorp/raft?

[divoxx]

Nice work. Is this still a WIP or complete? In a couple of tests I see a network abstraction (buildCluster and the network type) which seems to be for test purpose only but I don't really see a Interface nor an implementation of the actual network or transport.

I just glanced over the code though, might have missed something.

[jpfuentes2]

Grats team! See you at strangeloop, @bmizerany !

[divoxx]

I guess my question is: where and how will the network/transport be built and plugged against this?

[bmizerany]

@divoxx This is a work-in-progress. The network and storage is done outside of etcd/raft.

[bmizerany]

@divoxx like:

        node := raft.New(myid)
        for {   
                m := recvMessage()
                node.Step(m)

                saveState(node)

                ents := node.Next()
                applyEntriesToStateMachine(ents)

                out := node.Msgs()
                sendMessages(out)
        }

[bmizerany]

@peterbourgon The major points are stated in the body of this PR. To add to it: We want our implementation to be in Go and not have a hard dependency on a C library.

[peterbourgon]

@bmizerany Ah—I didn't know about the LMDB cgo thing! Thanks for that.

[divoxx]

@bmizerany interesting. I'm not sure what kind of transport are you guys going for but would be nice to have it as a standalone implementation (separate from etcd as well) to be also re-usable. Something like:

import (
    "net/http"
    "github.com/coreos/etcd/rafthttp"
)

func main() {
    // application sets up an m := http.ServeMux
    s := rafthttp.NewServer(m)
    log.Fatal(s.ListenAndServe())
}

[xiang90]

@divoxx We are working a clean and portable network/storage implement inside etcd. We do hope people can easily reuse that part of code or use it as an implementation reference.
I am not sure whether we will put in a standalone package or not.

[divoxx]

Perfect. Thanks @bmizerany and @xiangli-cmu

[bmizerany]

I think we need to change this comment, @unihorn. :)

[rogpeppe]

Just having a glance at this to see how it's getting on (I'm quite excited
by the project). A couple of comments in passing:

1. I expected that this would give me a working version:

% mkdir -p /home/rog/src/go/src/github.com/coreos/etcd
% cd /home/rog/src/go/src/github.com/coreos/etcd
% git init
% git remote add etcd-team git@github.com:etcd-team/etcd
% git fetch etcd-team
% git checkout raft
% go install ./...

but I get lots of errors like:
third_party/code.google.com/p/gogoprotobuf/protoc-gen-gogo/main.go:49:2: cannot find package "github.com/coreos/etcd/third_party/code.google.com/p/gogoprotobuf/plugin/defaultcheck" in any of:

I'm presuming there's a code generation step that I'm missing, but
it's not obvious what it is (genproto.sh also fails).

2. If github.com/coreos/etcd/raft is the principal raft package, it really wants
   some package docs, please. It's difficult to get an idea of how this is supposed to
   be used without them. The "Package raft is a generated protocol buffer
   package" comments aren't greatly informative :-) This applies to the
   other packages like snap and store too.

[xiang90]

@rogpeppe we have fixed the problem. you can try to build it again.

[rogpeppe]

I don't see any changes. The last commit I see on the raft branch is 0b2fdd6 from yesterday.

[xiang90]

@rogpeppe please try go build or go install inside etcd root directory.

[rogpeppe]

Ah, sorry. I expected all Go packages under that directory to be installable. My mistake.

[iwanttobepowerful]

@bmizerany r.bcastAppend() hi，what is the meaning of this line?And what will happen if we remove this line? thx!