Skip to content

Commit

Permalink
linear checker
Browse files Browse the repository at this point in the history
  • Loading branch information
ailidani authored and ailidani committed Jan 15, 2018
1 parent dd8e9a1 commit c98b493
Show file tree
Hide file tree
Showing 14 changed files with 490 additions and 78 deletions.
94 changes: 64 additions & 30 deletions README.md
View file
@@ -1,89 +1,123 @@
## What is Paxi?

## What is WPaxos?

**WPaxos** is a multileader Paxos protocol that provides low-latency and high-throughput consensus across wide-area network (WAN) deployments. Unlike statically partitioned multiple Paxos deployments, WPaxos perpetually adapts to the changing access locality through object stealing. Multiple concurrent leaders coinciding in different zones steal ownership of objects from each other using phase-1 of Paxos, and then use phase-2 to commit update-requests on these objects locally until they are stolen by other leaders. To achieve fast phase-2 commits, WPaxos adopts the flexible quorums idea in a novel manner, and appoints phase-2 acceptors to be close to their respective leaders.

WPaxos (WAN Paxos) paper (first version) can be found in https://arxiv.org/abs/1703.08905.
**Paxi** is the framework that implements WPaxos and other Paxos protocol variants. Paxi provides most of the elements that any Paxos implementation or replication protocol needs, including network communication, state machine of a key-value store, client API and multiple types of quorum systems.

*Warning*: Paxi project is still under heavy development, with more features and protocols to include. Paxi API may change too.

## What is Paxi?

**Paxi** is the framework that implements WPaxos and other Paxos protocol variants. Paxi provides most of the elements that any Paxos implementation or replication protocol needs, including network communication, state machine of a key-value store, client API and multiple types of quorum systems.
## What is WPaxos?

Warning: Paxi project is still under heavy development, with more features and protocols to include. Paxi API may change too.
**WPaxos** is a multileader Paxos protocol that provides low-latency and high-throughput consensus across wide-area network (WAN) deployments. Unlike statically partitioned multiple Paxos deployments, WPaxos perpetually adapts to the changing access locality through object stealing. Multiple concurrent leaders coinciding in different zones steal ownership of objects from each other using phase-1 of Paxos, and then use phase-2 to commit update-requests on these objects locally until they are stolen by other leaders. To achieve fast phase-2 commits, WPaxos adopts the flexible quorums idea in a novel manner, and appoints phase-2 acceptors to be close to their respective leaders.

WPaxos (WAN Paxos) paper (first version) can be found in https://arxiv.org/abs/1703.08905.

## What is included?

Algorithms:
- [x] Classical multi-Paxos
- [x] [Flexible Paxos](https://dl.acm.org/citation.cfm?id=3139656)
- [x] [WPaxos](https://arxiv.org/abs/1703.08905)
- [x] [EPaxos](https://dl.acm.org/citation.cfm?id=2517350)
- [x] KPaxos (Static partitioned Paxos)
- [ ] [Vertical Paxos](https://www.microsoft.com/en-us/research/wp-content/uploads/2009/08/Vertical-Paxos-and-Primary-Backup-Replication-.pdf)
- [ ] [WanKeeper](http://ieeexplore.ieee.org/abstract/document/7980095/)

Features:
- [x] Benchmarking
- [x] Linerizability checker
- [ ] Transactions
- [ ] Dynamic quorums
- [ ] Fault injection
- [ ] Linerizability checker


# How to build

1. Install [Go 1.9](https://golang.org/dl/).
2. [Download](https://github.com/wpaxos/paxi/archive/master.zip) WPaxos source code from GitHub page or use following command:
2. Use `go get` command or [Download](https://github.com/wpaxos/paxi/archive/master.zip) Paxi source code from GitHub page.
```
go get github.com/ailidani/paxi
```

3. Compile everything.
3. Compile everything from `paxi/bin` folder.
```
cd github.com/ailidani/paxi/bin
./build.sh
```

After compile, Golang will generate 4 executable files under `bin` folder.
* `master` is the easy way to distribute configurations to all replica nodes.
* `server` is one replica instance.
* `client` is a simple benchmark that generates read/write reqeust to servers.
* `cmd` is a command line tool to test Get/Set requests.
* `master` is the alternative way to distribute configurations to all replica nodes.


# How to run

Each executable file expects some parameters which can be seen by `-help` flag, e.g. `./master -help`.
Each executable file expects some parameters which can be seen by `-help` flag, e.g. `./server -help`.

1. There are two ways to manage the system configuration.

(1) Use a [configuration file](https://github.com/ailidani/paxi/blob/master/bin/config.json) with `-config FILE_PATH` option, default to "config.json" when omit.

1. Start master node with 6 replicas running WPaxos:
(2) Start a master node with 6 replica nodes running WPaxos:
```
./master.sh -n 6 -algorithm "wpaxos"
```

2. Start 6 servers with different zone id and node ids.
2. Start 6 servers with different ids in format of "ZONE_ID.NODE_ID".
```
./server -id 1.1 -master 127.0.0.1 &
./server -id 1.2 -master 127.0.0.1 &
./server -id 2.1 -master 127.0.0.1 &
./server -id 2.2 -master 127.0.0.1 &
./server -id 3.1 -master 127.0.0.1 &
./server -id 3.2 -master 127.0.0.1 &
./server -id 1.1 &
./server -id 1.2 &
./server -id 2.1 &
./server -id 2.2 &
./server -id 3.1 &
./server -id 3.2 &
```

3. Start benchmarking client with 10 threads, 1000 keys, 50 percent conflicting commands and run for 60 seconds in 1 round.
3. Start benchmarking client that connects to server ID 1.1 and benchmark parameters specified in [benchmark.json](https://github.com/ailidani/paxi/blob/master/bin/benchmark.json).
```
./client -id 1.1 -master 127.0.0.1 -bconfig benchmark.json
./client -id 1.1 -bconfig benchmark.json
```

The algorithms can also be running in **simulation** mode, where all nodes are running in one process and transport layer is replaced by Go channels. Check [`simulation.sh`](https://github.com/ailidani/paxi/blob/master/bin/simulation.sh) script on how to run.


# How to implement algorithms in Paxi

Replication algorithm in Paxi follows the message passing model, where several message types and their handle function are registered.
Replication algorithm in Paxi follows the message passing model, where several message types and their handle function are registered. We use [Paxos](https://github.com/ailidani/paxi/tree/master/paxos) as an example for our step-by-step tutorial.

1. Define messages, register with gob in `init()` function if using gob codec.
1. Define messages, register with gob in `init()` function if using gob codec. As show in [`msg.go`](https://github.com/ailidani/paxi/blob/master/paxos/msg.go).

2. Define handle function for each message type.
2. Define a `Replica` structure embeded with `paxi.Node` interface.
```go
type Replica struct {
paxi.Node
*Paxos
}
```

Define handle function for each message type. For example, to handle client `Request`
```go
func (r *Replica) handleRequest(m paxi.Request) {
if r.Config().Adaptive {
if r.Paxos.IsLeader() || r.Paxos.Ballot() == 0 {
r.Paxos.HandleRequest(m)
} else {
go r.Forward(r.Paxos.Leader(), m)
}
} else {
r.Paxos.HandleRequest(m)
}

}
```

3. Register the messages with their handle function using `Node.Register(interface{}, func())` interface.
3. Register the messages with their handle function using `Node.Register(interface{}, func())` interface in `Replica` constructor.

For sending messages, use `Send(to ID, msg interface{})`, `Broadcast(msg interface{})` functions in Node.Socket.
Replica use `Send(to ID, msg interface{})`, `Broadcast(msg interface{})` functions in Node.Socket to send messages.

For data-store related functions check db.go file.
For data-store related functions check `db.go` file.

For quorum types check quorum.go file.
For quorum types check `quorum.go` file.

Client uses a simple RESTful API to submit requests. GET method with URL "http://ip:port/key" will read the value of given key. POST method with URL "http://ip:port/key" and body as the value, will write the value to key.
4 changes: 2 additions & 2 deletions ballot_test.go
View file
Expand Up @@ -9,8 +9,8 @@ func TestBallot(t *testing.T) {
id := NewID(2, 1)
b := NewBallot(n, id)

b.Next()
b.Next()
b.Next(id)
b.Next(id)

if b.N() != n+2 {
t.Errorf("Ballot.N() %v != %v", b.N(), n+1)
Expand Down
79 changes: 47 additions & 32 deletions benchmarker.go
View file
Expand Up @@ -21,12 +21,13 @@ type DB interface {
var file = flag.String("bconfig", "benchmark.json", "benchmark configuration file")

type bconfig struct {
T int // total number of running time in seconds
N int // total number of requests
K int // key sapce
W int // percentage of writes
Concurrency int // number of simulated clients
Distribution string // distribution
T int // total number of running time in seconds
N int // total number of requests
K int // key sapce
W int // percentage of writes
Concurrency int // number of simulated clients
Distribution string // distribution
LinearizabilityCheck bool // run linearizability checker at the end of benchmark
// rounds int // repeat in many rounds sequentially

// random distribution
Expand All @@ -48,20 +49,21 @@ type bconfig struct {

func NewBenchmarkConfig() bconfig {
return bconfig{
T: 10,
N: 0,
K: 1000,
W: 100,
Concurrency: 1,
Distribution: "random",
Conflicts: 100,
Min: 0,
Mu: 0,
Sigma: 60,
Move: false,
Speed: 500,
Zipfian_s: 2,
Zipfian_v: 1,
T: 10,
N: 0,
K: 1000,
W: 100,
Concurrency: 1,
Distribution: "random",
LinearizabilityCheck: false,
Conflicts: 100,
Min: 0,
Mu: 0,
Sigma: 60,
Move: false,
Speed: 500,
Zipfian_s: 2,
Zipfian_v: 1,
}
}

Expand All @@ -88,6 +90,7 @@ func (c *bconfig) Save() error {
type Benchmarker struct {
db DB // read/write operation interface
bconfig
History

cwait sync.WaitGroup // wait for all clients to finish
latency []time.Duration // latency per operation for each round
Expand All @@ -98,11 +101,12 @@ func NewBenchmarker(db DB) *Benchmarker {
b := new(Benchmarker)
b.db = db
b.bconfig = NewBenchmarkConfig()
b.History = NewHistory()
return b
}

// Start starts the main logic of benchmarking
func (b *Benchmarker) Start() {
// Run starts the main logic of benchmarking
func (b *Benchmarker) Run() {
rand.Seed(time.Now().UTC().UnixNano())
r := rand.New(rand.NewSource(time.Now().UTC().UnixNano()))
b.zipf = rand.NewZipf(r, b.Zipfian_s, b.Zipfian_v, uint64(b.K))
Expand All @@ -111,14 +115,16 @@ func (b *Benchmarker) Start() {
if b.Move {
move := func() { b.Mu = float64(int(b.Mu+1) % b.K) }
stop = Schedule(move, time.Duration(b.Speed)*time.Millisecond)
defer close(stop)
}

b.latency = make([]time.Duration, 1000)
start_time := time.Now()
b.db.Init()

keys := make(chan int, 1000)
keys := make(chan int, b.Concurrency)
results := make(chan time.Duration, 1000)
defer close(results)
go b.collect(results)
for i := 0; i < b.Concurrency; i++ {
go b.worker(keys, results)
Expand All @@ -142,18 +148,21 @@ func (b *Benchmarker) Start() {

b.db.Stop()
end_time := time.Now()
close(keys)
stat := Statistic(b.latency)
stat.WriteFile("latency")
t := end_time.Sub(start_time)
log.Infof("Benchmark took %v\n", t)
log.Infof("Throughput %f\n", float64(len(b.latency))/t.Seconds())
log.Infoln(stat)

if b.Move {
stop <- true
if b.LinearizabilityCheck {
if b.History.Linearizable() {
log.Infoln("The execution is linearizable.")
} else {
log.Infoln("The execution is NOT linearizable.")
}
}
close(keys)
close(results)
}

// generates key based on distribution
Expand Down Expand Up @@ -185,15 +194,21 @@ func (b *Benchmarker) next() int {

func (b *Benchmarker) worker(keys <-chan int, results chan<- time.Duration) {
for k := range keys {
v := rand.Int()

s := time.Now()
var s time.Time
var e time.Time
if rand.Intn(100) < b.W {
v := rand.Int()
s = time.Now()
b.db.Write(k, v)
e = time.Now()
b.History.Add(k, v, nil, s.UnixNano(), e.UnixNano())
} else {
b.db.Read(k)
s = time.Now()
v := b.db.Read(k)
e = time.Now()
b.History.Add(k, nil, v, s.UnixNano(), e.UnixNano())
}
t := time.Now().Sub(s)
t := e.Sub(s)
results <- t
}
}
Expand Down
3 changes: 2 additions & 1 deletion bin/benchmark.json
View file
Expand Up @@ -2,9 +2,10 @@
"T": 60,
"N": 0,
"K": 1000,
"W": 100,
"W": 50,
"Concurrency": 1,
"Distribution": "random",
"LinearizabilityCheck": true,
"Conflicts": 100,
"Min": 0,
"Mu": 0,
Expand Down

0 comments on commit c98b493

Please sign in to comment.