Merge pull request #171 from liftbridge-io/config

Use yaml as the only configuration file format

documentation/configuration.md

@@ -10,7 +10,7 @@ over settings in the configuration file.
 The configuration file is passed in using the `--config` flag:
 
 ```shell
-$ liftbridge --config liftbridge.conf
+$ liftbridge --config liftbridge.yaml
 ```
 
 To get a full list of the CLI flags, use the `--help` flag:
@@ -49,49 +49,56 @@ GLOBAL OPTIONS:
 
 ## Configuration File Format
 
-The configuration file format borrows from the [NATS configuration
-format](https://nats.io/documentation/server/gnatsd-config/). It is a flexible
-format that combines the best of traditional configuration formats and newer
-styles such as JSON and YAML.
+The configuration file uses a YAML format. The configuration settings are
+described below. Settings follow a hierarchical pattern delimited by periods.
+The full, flattened name or nested names can be used. For example:
 
-The config file format supports the following syntax:
+```yaml
+logging.level: debug
+logging.recovery: true
+logging.raft: true
+```
+
+The above configuration is equivalent to:
 
-- Mixed Arrays: `[...]`
-- Nested Maps: `{...}`
-- Multiple comment types: `#` and `//`
-- Key value assigments using:
-  - Equals sign (`foo = 2`)
-  - Colon (`foo: 2`)
-  - Whitespace (`foo 2`)
-- Maps can be assigned with no key separator
-- Semicolons as value terminators in key/value assignments are optional
+```yaml
+logging:
+  level: debug
+  recovery: true
+  raft: true
+```
 
 ## Example Configuration File
 
 An example configuration file is shown below.
 
-```plaintext
+```yaml
+---
 listen: localhost:9293
 data.dir: /tmp/liftbridge/server-2
-log.level: debug
+
+# Configure logging.
+logging:
+  level: debug
+  raft: true
 
 # Define NATS cluster to connect to.
-nats {
-    servers: ["nats://localhost:4300", "nats://localhost:4301"]
-}
+nats.servers:
+  - nats://localhost:4300
+  - nats://localhost:4301
 
-# Specify message log settings.
-log {
-    retention.max.age: "24h"
-}
+# Specify message stream settings.
+streams:
+  retention.max:
+    age: 24h
+    messages: 100
+  compact.enabled: true
 
 # Specify cluster settings.
-clustering {
-    server.id: server-2
-    raft.logging: true
-    raft.bootstrap.seed: true
-    replica.max.lag.time: "20s"
-}
+clustering:
+  server.id: server-2
+  raft.bootstrap.seed: true
+  replica.max.lag.time: 20s
 ```
 
 ## Configuration Settings
@@ -106,17 +113,18 @@ the setting in the configuration file and the CLI flag if it exists.
 | port | port | The server port that is advertised to clients. See `host` for more information on how this behaves. | int | 9292 | |
 | tls.key | tls-key | The private key file for server certificate. This must be set in combination with `tls.cert` to enable TLS. | string | |
 | tls.cert | tls-cert | The server certificate file. This must be set in combination with `tls.key` to enable TLS. | string | |
-| tls.client.auth | tls-client-auth | Enforce client-side authentication via certificate. | bool | false |
+| tls.client.auth.enabled | tls-client-auth | Enforce client-side authentication via certificate. | bool | false |
 | tls.client.auth.ca | tls-client-auth-ca | The CA certificate file to use when authenticating clients. | string | |
-| log.level | level | The logging level. | string | info | [debug, info, warn, error] |
-| log.recovery | | Log messages resulting from the replay of the Raft log on server recovery. | bool | false | |
+| logging.level | level | The logging level. | string | info | [debug, info, warn, error] |
+| logging.recovery | | Log messages resulting from the replay of the Raft log on server recovery. | bool | false | |
+| logging.raft | | Enables logging in the Raft subsystem. | bool | false | |
 | data.dir | data-dir | The directory to store data in. | string | /tmp/liftbridge/namespace | |
 | batch.max.messages | | The maximum number of messages to batch when writing to disk. | int | 1024 |
-| batch.wait.time | | The time to wait to batch more messages when writing to disk. | duration | 0 | |
+| batch.max.time | | The maximum time to wait to batch more messages when writing to disk. | duration | 0 | |
 | metadata.cache.max.age | | The maximum age of cached broker metadata. | duration | 2m | |
 | nats | | NATS configuration. | map | | [See below](#nats-configuration-settings) |
-| log | | Stream write-ahead log configuration. | map | | [See below](#log-configuration-settings) |
-| clustering | | Broker cluster configuration. | map | | [See below](#cluster-configuration-settings) |
+| streams | | Write-ahead log configuration for message streams. | map | | [See below](#streams-configuration-settings) |
+| clustering | | Broker cluster configuration. | map | | [See below](#clustering-configuration-settings) |
 
 ### NATS Configuration Settings
 
@@ -129,21 +137,21 @@ the configuration file.
 | user | | Username to use to connect to NATS servers. | string | | |
 | password | | Password to use to connect to NATS servers. | string | | |
 
-### Log Configuration Settings
+### Streams Configuration Settings
 
-Below is the list of the configuration settings for the `log` part of
-the configuration file.
+Below is the list of the configuration settings for the `streams` part of the
+configuration file.
 
 | Name | Flag | Description | Type | Default | Valid Values |
 |:----|:----|:----|:----|:----|:----|
 | retention.max.bytes | | The maximum size a stream's log can grow to, in bytes, before we will discard old log segments to free up space. A value of 0 indicates no limit. | int64 | 0 | |
 | retention.max.messages | | The maximum size a stream's log can grow to, in number of messages, before we will discard old log segments to free up space. A value of 0 indicates no limit. | int64 | 0 | |
 | retention.max.age | | The TTL for stream log segment files, after which they are deleted. A value of 0 indicates no TTL. | duration | 168h | |
 | cleaner.interval | | The frequency to check if a new stream log segment file should be rolled and whether any segments are eligible for deletion based on the retention policy or compaction if enabled. | duration | 5m | |
-| log.roll.time | | The maximum time before a new stream log segment is rolled out. A value of 0 means new segments will only be rolled when `segment.max.bytes` is reached. Retention is always done a file at a time, so a larger value means fewer files but less granular control over retention. | duration | value of `retention.max.age` | |
 | segment.max.bytes | | The maximum size of a single stream log segment file in bytes. Retention is always done a file at a time, so a larger segment size means fewer files but less granular control over retention. | int64 | 268435456 | |
-| compact | | Enables stream log compaction. Compaction works by retaining only the latest message for each key and discarding older messages. The frequency in which compaction runs is controlled by `cleaner.interval`. | bool | false | |
-| compact.max.goroutines | | The maximum number of concurrent goroutines to use for compaction on a stream log (only applicable if `compact` is enabled). | int | 10 | |
+| segment.max.age | | The maximum time before a new stream log segment is rolled out. A value of 0 means new segments will only be rolled when `segment.max.bytes` is reached. Retention is always done a file at a time, so a larger value means fewer files but less granular control over retention. | duration | value of `retention.max.age` | |
+| compact.enabled | | Enables stream log compaction. Compaction works by retaining only the latest message for each key and discarding older messages. The frequency in which compaction runs is controlled by `cleaner.interval`. | bool | false | |
+| compact.max.goroutines | | The maximum number of concurrent goroutines to use for compaction on a stream log (only applicable if `compact.enabled` is `true`). | int | 10 | |
 
 ### Clustering Configuration Settings
 
@@ -159,7 +167,6 @@ the configuration file.
 | raft.cache.size | | The number of Raft logs to hold in memory for quick lookup. | int | 512 | |
 | raft.bootstrap.seed | raft-bootstrap-seed | Bootstrap the Raft cluster by electing self as leader if there is no existing state. If this is enabled, `raft.bootstrap.peers` should generally not be used, either on this node or peer nodes, since cluster topology is not being explicitly defined. Instead, peers should be started without bootstrap flags which will cause them to automatically discover the bootstrapped leader and join the cluster. | bool | false | |
 | raft.bootstrap.peers | raft-bootstrap-peers | Bootstrap the Raft cluster with the provided list of peer IDs if there is no existing state. This should generally not be used in combination with `raft.bootstrap.seed` since it is explicitly defining cluster topology and the configured topology will elect a leader. Note that once the cluster is established, new nodes can join without setting bootstrap flags since they will automatically discover the elected leader and join the cluster. | list | | |
-| raft.logging | | Enables logging in the Raft subsystem. | bool | false | |
 | replica.max.lag.time | | If a follower hasn't sent any replication requests or hasn't caught up to the leader's log end offset for at least this time, the leader will remove the follower from ISR. | duration | 15s | |
 | replica.max.leader.timeout | | If a leader hasn't sent any replication responses for at least this time, the follower will report the leader to the controller. If a majority of the replicas report the leader, a new leader is selected by the controller. | duration | 15s | |
 | replica.max.idle.wait | | The maximum amount of time a follower will wait before making a replication request once the follower is caught up with the leader. This value should always be less than `replica.max.lag.time` to avoid frequent shrinking of ISR for low-throughput streams. | duration | 10s | |

go.mod

@@ -19,6 +19,7 @@ require (
 	github.com/nsip/gommap v0.0.0-20181229045655-f7881c3a959f
 	github.com/pkg/errors v0.9.1
 	github.com/sirupsen/logrus v1.4.2
+	github.com/spf13/viper v1.6.2
 	github.com/stretchr/testify v1.4.0
 	github.com/urfave/cli v1.22.3
 	google.golang.org/grpc v1.28.0