Allow config options in file as yaml mappings

Replaces the `ff` package with a custom implementation that uses Viper
to parse the config file and sets the values for the `flag.Flag`s.

By design the ff package doesn't allow flag subsets or nested maps inside
the config yaml and all config options must be explicitely defined as
`flag.Flag`. This limits the ability to specify more dynamic config options
like lists or maps of user defined rules (Ex: metric rollups).

Viper provides the flexibility we require for parsing the config file, and
unmarshaling directly into more complex datastructures. To define how to
unmarshal a subtree of the configuration into a datastructure via
unmarshalRule. An unmarshalRule for the dataset config was added as:

```
withUnmarshalRules(
  []unmarshalRule{{"startup.dataset", &dataset.Config}}
)
```

Given a config file that contains the subtree:

```
startup:
  dataset:
    metrics:
      ...
    traces:
      ...
```

The subtree `startup.dataset` will be unmarshal into `&dataset.Config`
by Viper using the github.com/mitchellh/mapstructure package.

Viper can access a nested field by passing a `.` delimited path of keys.
This change also allows us to format our configuration files to use
yaml mappings instead of long key names with `.`, thus maintaining
backwards compatibility:

```
web.listen-address: localhost:9201
web.auth.password: my-password
web.auth.username: promscale
```

Can be rewritten as:

```
web:
  listen-address: localhost:9201
  auth:
    password: my-password
    username: promscale
```

CHANGELOG.md

@@ -18,6 +18,10 @@ We use the following categories for changes:
 ### Added
 - Alerts from promscale monitoring mixin are groupped also by namespace label [#1714]
 - Added a new family of metrics tracking database maintenance jobs durations and failures [#1745]
+- Allow config options in file to be set as yaml mappings [#1737]
+- Add `startup.dataset` option in the config file for the dataset
+  configuration. Supersedes `startup.dataset.config` which accepts a string
+  instead of a mapping [#1737]
 
 ### Changed
 - Reduced the verbosity of the logs emitted by the vacuum engine [#1715]

docs/dataset.md

@@ -2,28 +2,19 @@
 
 Promscale stores some configuration information in the Postgres database which it is connected to. We call this configuration the *Promscale dataset configuration*. Promscale accepts an option to set the dataset values. This document describes its format and mechanics.
 
-Setup is done by using the `-startup.dataset.config` flag which should contain this configuration structure in YAML format
-
-Example usage:
-
-```bash
-promscale -startup.dataset.config=$'metrics:\n  default_chunk_interval: 6h'
-```
-
-The expected format is YAML.
-
-The YAML values are easier to set when using `config.yaml` instead of the cli flags. This would look as follows:
+Setup is done via the `config.yaml` under `startup.dataset`:
 
 ```yaml
-startup.dataset.config: |
-  metrics:
-    default_chunk_interval: 6h
-    compress_data: true
-    ha_lease_refresh: 10s
-    ha_lease_timeout: 1m
-    default_retention_period: 90d
-  traces:
-    default_retention_period: 30d
+startup:
+  dataset:
+    metrics:
+      default_chunk_interval: 6h
+      compress_data: true
+      ha_lease_refresh: 10s
+      ha_lease_timeout: 1m
+      default_retention_period: 90d
+    traces:
+      default_retention_period: 30d
 ```
 
 Note: Any configuration omitted from the configuration structure will be set to its default value.
@@ -32,9 +23,9 @@ Note: Any configuration omitted from the configuration structure will be set to
 
 | Section | Setting                  | Type     | Default | Description                                                                                                     |
 |:--------|:-------------------------|:--------:|:-------:|:----------------------------------------------------------------------------------------------------------------|
-| metric  | default_chunk_interval   | duration |   8h    | Chunk interval used to create hypertable chunks that store the metric data                                      |
-| metric  | compress_data            |   bool   |  true   | Boolean setting to turn on or off compression of metric data                                                    |
-| metric  | ha_lease_refresh         | duration |   10s   | High availability lease refresh duration, period after which the lease will be refreshed                        |
-| metric  | ha_lease_timeout         | duration |   1m    | High availability lease timeout duration, period after which the lease will be lost in case it wasn't refreshed |
-| metric  | default_retention_period | duration |   90d   | Retention period for metric data, all data older than this period will be dropped                               |
+| metrics | default_chunk_interval   | duration |   8h    | Chunk interval used to create hypertable chunks that store the metric data                                      |
+| metrics | compress_data            |   bool   |  true   | Boolean setting to turn on or off compression of metric data                                                    |
+| metrics | ha_lease_refresh         | duration |   10s   | High availability lease refresh duration, period after which the lease will be refreshed                        |
+| metrics | ha_lease_timeout         | duration |   1m    | High availability lease timeout duration, period after which the lease will be lost in case it wasn't refreshed |
+| metrics | default_retention_period | duration |   90d   | Retention period for metric data, all data older than this period will be dropped                               |
 | traces  | default_retention_period | duration |   90d   | Retention period for tracing data, all data older than this period will be dropped                              |

pkg/dataset/config.go

@@ -36,22 +36,22 @@ var (
 
 // Config represents a dataset config.
 type Config struct {
-	Metrics `yaml:"metrics"`
-	Traces  `yaml:"traces"`
+	Metrics
+	Traces
 }
 
 // Metrics contains dataset configuration options for metrics data.
 type Metrics struct {
-	ChunkInterval   DayDuration `yaml:"default_chunk_interval"`
-	Compression     *bool       `yaml:"compress_data"` // Using pointer to check if the the value was set.
-	HALeaseRefresh  DayDuration `yaml:"ha_lease_refresh"`
-	HALeaseTimeout  DayDuration `yaml:"ha_lease_timeout"`
-	RetentionPeriod DayDuration `yaml:"default_retention_period"`
+	ChunkInterval   DayDuration `mapstructure:"default_chunk_interval" yaml:"default_chunk_interval"`
+	Compression     *bool       `mapstructure:"compress_data" yaml:"compress_data"` // Using pointer to check if the the value was set.
+	HALeaseRefresh  DayDuration `mapstructure:"ha_lease_refresh" yaml:"ha_lease_refresh"`
+	HALeaseTimeout  DayDuration `mapstructure:"ha_lease_timeout" yaml:"ha_lease_timeout"`
+	RetentionPeriod DayDuration `mapstructure:"default_retention_period" yaml:"default_retention_period"`
 }
 
 // Traces contains dataset configuration options for traces data.
 type Traces struct {
-	RetentionPeriod DayDuration `yaml:"default_retention_period"`
+	RetentionPeriod DayDuration `mapstructure:"default_retention_period" yaml:"default_retention_period"`
 }
 
 // NewConfig creates a new dataset config based on the configuration YAML contents.

pkg/dataset/duration.go

@@ -5,8 +5,11 @@ package dataset
 
 import (
 	"fmt"
+	"reflect"
 	"strings"
 	"time"
+
+	"github.com/mitchellh/mapstructure"
 )
 
 const (
@@ -67,3 +70,28 @@ func handleDays(s []byte) (time.Duration, error) {
 func (d DayDuration) String() string {
 	return time.Duration(d).String()
 }
+
+// StringToDayDurationHookFunc returns a mapstructure.DecodeHookFunc that
+// converts strings to DayDuration.
+func StringToDayDurationHookFunc() mapstructure.DecodeHookFunc {
+	return func(
+		f reflect.Type,
+		t reflect.Type,
+		data interface{}) (interface{}, error) {
+		if f.Kind() != reflect.String {
+			return data, nil
+		}
+
+		var d DayDuration
+
+		if t != reflect.TypeOf(d) {
+			return data, nil
+		}
+
+		err := d.UnmarshalText([]byte(data.(string)))
+		if err != nil {
+			return nil, err
+		}
+		return DayDuration(d), nil
+	}
+}

pkg/runner/client.go

@@ -165,11 +165,13 @@ func CreateClient(r prometheus.Registerer, cfg *Config) (*pgclient.Client, error
 		cfg.APICfg.MultiTenancy = multiTenancy
 	}
 
-	if cfg.DatasetConfig != "" {
-		err = ApplyDatasetConfig(conn, cfg.DatasetConfig)
-		if err != nil {
-			return nil, fmt.Errorf("error applying dataset configuration: %w", err)
-		}
+	if (cfg.DatasetCfg != dataset.Config{}) {
+		err = cfg.DatasetCfg.Apply(conn)
+	} else if cfg.DatasetConfig != "" {
+		err = applyDatasetConfig(conn, cfg.DatasetConfig)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("error applying dataset configuration: %w", err)
 	}
 
 	// client has to be initiated after migrate since migrate
@@ -226,7 +228,7 @@ func isBGWLessThanDBs(conn *pgx.Conn) (bool, error) {
 	return false, nil
 }
 
-func ApplyDatasetConfig(conn *pgx.Conn, cfgFilename string) error {
+func applyDatasetConfig(conn *pgx.Conn, cfgFilename string) error {
 	cfg, err := dataset.NewConfig(cfgFilename)
 	if err != nil {
 		return err