Skip to content

Commit

Permalink
Resolve and re-set BindAddr to propagate an unique IP address
Browse files Browse the repository at this point in the history
  • Loading branch information
@buraksezer
buraksezer committed Apr 26, 2020
1 parent 18f3c0c commit d5f5de6
Show file tree
Hide file tree
Showing 9 changed files with 241 additions and 280 deletions.
162 changes: 27 additions & 135 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,9 @@ Olric is in early stages of development. The package API and client protocol may
* [Serialization](#serialization)
* [Golang Client](#golang-client)
* [Configuration](#configuration)
* [Embedded Member Mode](#embedded-member-mode)
* [Client-Server Mode](#client-server-mode)
* [Network Configuration](#network-configuration)
* [Service discovery](#service-discovery)
* [Architecture](#architecture)
* [Overview](#overview)
Expand Down Expand Up @@ -807,152 +810,42 @@ configuration parameters, see [Olric documentation on GoDoc.org](https://godoc.o

## Configuration

[memberlist configuration](https://godoc.org/github.com/hashicorp/memberlist#Config) can be tricky and the default configuration set should be tuned for your environment. A detailed deployment and configuration guide will be prepared before stable release.
You should feel free to ask any questions about configuration and integration. Please see [Support](#support) section.

Please take a look at [Config section at godoc.org](https://godoc.org/github.com/buraksezer/olric#Config)
### Embedded-Member Mode

It's generally good to use `config.New` function to get the default configuration. It takes `local`, `lan` and `wan` parameters.
Please see the [documentation](https://godoc.org/github.com/buraksezer/olric/config#New) to get more information.

Here is a sample configuration for a cluster with two hosts:
Olric provides a function to generate default configuration to use in embedded-member mode:

```go
package main

import (
"context"
"fmt"
"log"
"net"
"reflect"
"strconv"
"time"

"github.com/buraksezer/olric"
"github.com/buraksezer/olric/config"
)

const (
bindAddr string = "0.0.0.0"
peerPortOne int = 5555
peerPortTwo int = 6666
)

func config1() *config.Config {
c := config.New("local")
// overwrite default values
c.BindPort = 3320
c.Name = net.JoinHostPort(bindAddr, strconv.Itoa(c.BindPort))

// Add the peer
c.MemberlistConfig.BindPort = peerPortOne
peerTwo := net.JoinHostPort(bindAddr, strconv.Itoa(peerPortTwo))
c.Peers = []string{peerTwo}
return c
}

func config2() *config.Config {
c := config.New("local")

// overwrite default values
c.BindPort = 3322
c.Name = net.JoinHostPort(bindAddr, strconv.Itoa(c.BindPort))

// Add the peer
c.MemberlistConfig.BindPort = peerPortTwo
peerOne := net.JoinHostPort(bindAddr, strconv.Itoa(peerPortOne))
c.Peers = []string{peerOne}
return c
}

func main() {
c1 := config1()
db1, err := olric.New(c1)
if err != nil {
log.Fatalf("Failed to create Olric object: %v", err)
}
go func() {
// Call Start at background. It's a blocker call.
err = db1.Start()
if err != nil {
log.Fatalf("Failed to call Start: %v", err)
}
}()

c2 := config2()
// This creates a single-node Olric cluster. It's good enough for experimenting.
db2, err := olric.New(c2)
if err != nil {
log.Fatalf("Failed to create Olric object: %v", err)
}
import "github.com/buraksezer/olric/config"
...
c := config.New("local")
```

go func() {
// Call Start at background. It's a blocker call.
err = db2.Start()
if err != nil {
log.Fatalf("Failed to call Start: %v", err)
}
}()
The `New` function takes a parameter called `env`. It denotes the network environment and consumed by [hashicorp/memberlist](https://github.com/hashicorp/memberlist).
Default configuration is good enough for distributed caching scenario. In order to see all configuration parameters, please take a look at [this](https://godoc.org/github.com/buraksezer/olric/config).

// You can use `config.Started` callback function to get notified about a server start
// for the sake of simplicity, we just call time.After for some time.
fmt.Println("Awaiting for background goroutines")
<-time.After(time.Second)
See [Sample Code](#sample-code) section for an introduction.

// Put 10 items into the DMap object.
dm, err := db1.NewDMap("bucket-of-arbitrary-items")
if err != nil {
log.Fatalf("Failed to call NewDMap: %v", err)
}
### Client-Server Mode

fmt.Println("##")
fmt.Println("Operations on a DMap instance:")
err = dm.Put("string-key", "buraksezer")
if err != nil {
log.Fatalf("Failed to call Put: %v", err)
}
stringValue, err := dm.Get("string-key")
if err != nil {
log.Fatalf("Failed to call Get: %v", err)
}
fmt.Printf("Value for string-key: %v, reflect.TypeOf: %s\n", stringValue, reflect.TypeOf(stringValue))
Olric provides **olricd** to implement client-server mode. olricd gets a YAML file for the configuration. The most basic functionality of olricd is that
translating YAML configuration into Olric's configuration struct. A sample `olricd.yaml` file is being provided [here](https://github.com/buraksezer/olric/blob/master/cmd/olricd/olricd.yaml).

err = dm.Put("uint64-key", uint64(1988))
if err != nil {
log.Fatalf("Failed to call Put: %v", err)
}
uint64Value, err := dm.Get("uint64-key")
if err != nil {
log.Fatalf("Failed to call Get: %v", err)
}
fmt.Printf("Value for uint64-key: %v, reflect.TypeOf: %s\n", uint64Value, reflect.TypeOf(uint64Value))
### Network Configuration

err = dm.Put("nil-key", nil)
if err != nil {
log.Fatalf("Failed to call Put: %v", err)
}
nilValue, err := dm.Get("nil-key")
if err != nil {
log.Fatalf("Failed to call Get: %v", err)
}
fmt.Printf("Value for nil-key: %v\n", nilValue)
fmt.Println("##")
In an Olric instance, there are two different TCP servers. One for Olric, and the other one is for memberlist. `BindAddr` is very
critical to deploy a healthy Olric node. There are different scenarios:

// Don't forget the call Shutdown when you want to leave the cluster.
ctx1, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = db1.Shutdown(ctx1)
if err != nil {
log.Printf("Failed to shutdown Olric: %v", err)
}
* You can freely set a domain name or IP address as `BindAddr` for both Olric and memberlist. Olric will resolve and use it to bind.
* You can freely set `localhost`, `127.0.0.1` or `::1` as `BindAddr` in development environment for both Olric and memberlist.
* You can freely set `0.0.0.0` as `BindAddr` for both Olric and memberlist. Olric will pick an IP address, if there is any.
* If you don't set `BindAddr`, hostname will be used, and it will be resolved to get a valid IP address.
* You can set a network interface by using `Config.Interface` and `Config.MemberlistInterface` fields. Olric will find an appropriate IP address for the given interfaces, if there is any.
* You can set both `BindAddr` and interface parameters. In this case Olric will ensure that `BindAddr` is available on the given interface.

// Don't forget the call Shutdown when you want to leave the cluster.
ctx2, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = db2.Shutdown(ctx2)
if err != nil {
log.Printf("Failed to shutdown Olric: %v", err)
}
}
```
You should know that Olric needs a single and stable IP address to function properly. If you don't know the IP address of the host at the deployment time,
you can set `BindAddr` as `0.0.0.0`. Olric will very likely to find an IP address for you.

### Service Discovery

Expand Down Expand Up @@ -1218,7 +1111,6 @@ func main() {

<-ctx.Done()

// Put 10 items into the DMap object.
dm, err := db.NewDMap("bucket-of-arbitrary-items")
if err != nil {
log.Fatalf("olric.NewDMap returned an error: %v", err)
Expand Down
2 changes: 1 addition & 1 deletion cmd/olricd/olricd.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ olricd:
serializer: "msgpack"
keepAlivePeriod: "300s"
requestTimeout: "5s"
partitionCount: 71
partitionCount: 13
replicaCount: 1
writeQuorum: 1
readQuorum: 1
Expand Down
2 changes: 2 additions & 0 deletions cmd/olricd/server/config.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ type olricd struct {
Name string `yaml:"name"`
BindAddr string `yaml:"bindAddr"`
BindPort int `yaml:"bindPort"`
Interface string `yaml:"interface"`
ReplicationMode int `yaml:"replicationMode"`
PartitionCount uint64 `yaml:"partitionCount"`
LoadFactor float64 `yaml:"loadFactor"`
Expand All @@ -58,6 +59,7 @@ type memberlist struct {
Environment string `yaml:"environment"` // required
BindAddr string `yaml:"bindAddr"` // required
BindPort int `yaml:"bindPort"` // required
Interface string `yaml:"interface"`
EnableCompression *bool `yaml:"enableCompression"`
JoinRetryInterval string `yaml:"joinRetryInterval"` // required
MaxJoinAttempts int `yaml:"maxJoinAttempts"` // required
Expand Down
8 changes: 1 addition & 7 deletions cmd/olricd/server/memberlist.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ import (
"time"

"github.com/buraksezer/olric/config"
"github.com/buraksezer/olric/internal/network"
m "github.com/hashicorp/memberlist"
)

Expand Down Expand Up @@ -98,13 +97,8 @@ func newMemberlistConf(c *Config) (*m.Config, error) {

if c.Memberlist.AdvertiseAddr != nil {
mc.AdvertiseAddr = *c.Memberlist.AdvertiseAddr
} else {
advertiseAddr, err := network.ParseOrLookupIP(mc.BindAddr)
if err != nil {
return nil, err
}
mc.AdvertiseAddr = advertiseAddr.String()
}

if c.Memberlist.AdvertisePort != nil {
mc.AdvertisePort = *c.Memberlist.AdvertisePort
} else {
Expand Down
54 changes: 28 additions & 26 deletions cmd/olricd/server/server.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -153,32 +153,34 @@ func New(c *Config) (*Olricd, error) {

s.log = log.New(logOutput, "", log.LstdFlags)
s.config = &config.Config{
Name: c.Olricd.Name, // deprecated, it will be removed in v0.3.0
BindAddr: c.Olricd.BindAddr,
BindPort: c.Olricd.BindPort,
ServiceDiscovery: c.ServiceDiscovery,
MemberlistConfig: mc,
LogLevel: c.Logging.Level,
JoinRetryInterval: joinRetryInterval,
MaxJoinAttempts: c.Memberlist.MaxJoinAttempts,
Peers: c.Memberlist.Peers,
PartitionCount: c.Olricd.PartitionCount,
ReplicaCount: c.Olricd.ReplicaCount,
WriteQuorum: c.Olricd.WriteQuorum,
ReadQuorum: c.Olricd.ReadQuorum,
ReplicationMode: c.Olricd.ReplicationMode,
ReadRepair: c.Olricd.ReadRepair,
LoadFactor: c.Olricd.LoadFactor,
MemberCountQuorum: c.Olricd.MemberCountQuorum,
Logger: s.log,
LogOutput: logOutput,
LogVerbosity: c.Logging.Verbosity,
Hasher: hasher.NewDefaultHasher(),
Serializer: sr,
KeepAlivePeriod: keepAlivePeriod,
RequestTimeout: requestTimeout,
Cache: cacheConfig,
TableSize: c.Olricd.TableSize,
Name: c.Olricd.Name, // deprecated, it will be removed in v0.3.0
BindAddr: c.Olricd.BindAddr,
BindPort: c.Olricd.BindPort,
Interface: c.Olricd.Interface,
ServiceDiscovery: c.ServiceDiscovery,
MemberlistInterface: c.Memberlist.Interface,
MemberlistConfig: mc,
LogLevel: c.Logging.Level,
JoinRetryInterval: joinRetryInterval,
MaxJoinAttempts: c.Memberlist.MaxJoinAttempts,
Peers: c.Memberlist.Peers,
PartitionCount: c.Olricd.PartitionCount,
ReplicaCount: c.Olricd.ReplicaCount,
WriteQuorum: c.Olricd.WriteQuorum,
ReadQuorum: c.Olricd.ReadQuorum,
ReplicationMode: c.Olricd.ReplicationMode,
ReadRepair: c.Olricd.ReadRepair,
LoadFactor: c.Olricd.LoadFactor,
MemberCountQuorum: c.Olricd.MemberCountQuorum,
Logger: s.log,
LogOutput: logOutput,
LogVerbosity: c.Logging.Verbosity,
Hasher: hasher.NewDefaultHasher(),
Serializer: sr,
KeepAlivePeriod: keepAlivePeriod,
RequestTimeout: requestTimeout,
Cache: cacheConfig,
TableSize: c.Olricd.TableSize,
}
return s, nil
}
Expand Down
38 changes: 13 additions & 25 deletions config/config.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,10 @@ const (
)

const (
// DefaultPort is for Olric
DefaultPort = 3320

// DefaultDiscoveryPort is for memberlist
DefaultDiscoveryPort = 3322

// DefaultPartitionCount denotes default partition count in the cluster.
Expand Down Expand Up @@ -178,22 +180,20 @@ type CacheConfig struct {

// Config is the configuration to create a Olric instance.
type Config struct {
// Interface denotes a binding interface. It can be used instead of BindAddr if the interface is known but not the address.
// If both are provided, then Olric verifies that the interface has the bind address that is provided.
Interface string

// LogVerbosity denotes the level of message verbosity. The default value is 3. Valid values are between 1 to 6.
LogVerbosity int32

// Default LogLevel is DEBUG. Valid ones: "DEBUG", "WARN", "ERROR", "INFO"
LogLevel string

// Name of this node in the cluster. This must be unique in the cluster. If this is not set,
// Olric will set it to the hostname of the running machine. Example: node1.my-cluster.net
//
// Name is also used by the TCP server as Addr. It should be an IP address or domain name of the server.
Name string

// Addr to bind
// BindAddr denotes the address that Olric will bind to for communication with other Olric nodes.
BindAddr string

// Port to bind
// BindPort denotes the address that Olric will bind to for communication with other Olric nodes.
BindPort int

// KeepAlivePeriod denotes whether the operating system should send keep-alive messages on the connection.
Expand Down Expand Up @@ -267,6 +267,11 @@ type Config struct {

ServiceDiscovery map[string]interface{}

// Interface denotes a binding interface. It can be used instead of memberlist.Config.BindAddr if the interface is
// known but not the address. If both are provided, then Olric verifies that the interface has the bind address that
// is provided.
MemberlistInterface string

// MemberlistConfig is the memberlist configuration that Olric will
// use to do the underlying membership management and gossip. Some
// fields in the MemberlistConfig will be overwritten by Olric no
Expand Down Expand Up @@ -413,23 +418,6 @@ func (c *Config) Sanitize() error {
c.BindPort = DefaultPort
}

// TODO: Config.Name is deprecated and it will be removed in v0.3.0
if c.Name == "" {
c.Name = c.BindAddr + ":" + strconv.Itoa(c.BindPort)
} else {
c.Logger.Printf("[WARN] Config.Name is deprecated. Please use BindAddr and BindPort instead of Config.name")
host, _port, err := net.SplitHostPort(c.Name)
if err != nil {
return err
}
port, err := strconv.Atoi(_port)
if err != nil {
return err
}
c.BindAddr = host
c.BindPort = port
}

if c.LoadFactor == 0 {
c.LoadFactor = DefaultLoadFactor
}
Expand Down
Loading

0 comments on commit d5f5de6

Please sign in to comment.