Skip to content

Commit

Permalink
updated doc
Browse files Browse the repository at this point in the history 
Signed-off-by: Frederic BIDON <fredbi@yahoo.com>
  • Loading branch information
@fredbi
fredbi committed Sep 21, 2023
1 parent 1f104eb commit c2bc4d7
Show file tree
Hide file tree
Showing 3 changed files with 246 additions and 70 deletions.
260 changes: 204 additions & 56 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,40 +6,69 @@

# go-cli

This repo exposes a few utilities to build command-line utilities and manage configurations on top of
3 great librabries: `cobra`, `viper` and `pflag`. These libraries come in handy to build "12-factors" applications.
This repo exposes a few utilities to
(i) [build command-line utilities](#CLI) with
(ii) [flexible configurations](#Configuration)
on top of these 3 great libraries:
`github.com/spf13/cobra`, `github.com/spf13/viper` and `github.com/spf13/pflag`.

This is based on the seminal work by @casualjim. I am grateful to him for his much inspiring code.
The config part is based on some seminal past work by @casualjim. I am grateful to him for his much inspiring code.

**TL,DR**: this is not yet another CLI-building library, rather a mere wrapper on top of `cobra`
to use that great lib with a better style (IMHO).

## CLI

### Goals

The `cli` packages proposes an approach to building command-line binaries on top of the very rich `github.com/spf13/cobra` package.
The `cli` packages proposes an approach to building command-line binaries on top of `github.com/spf13/cobra`.

> There are a few great existing libraries around to build a CLI (see below, a few that I like).
> `cobra` stands out as the richest and most flexible,
> as CLIs are entirely built programmatically.
`cobra` is great, but building CLIs again and again, I came to identify a few repetitive boiler-plate patterns.

So this module reflects my opinions about how to build more elegant CLIs, with more expressive code and less tinkering.

Feedback is always welcome, as opinions may evolve over time...
Feel free to post issues to leave your comments and/or proposals.

#### Desirable features

* a typical CLI should interact easily with config files (see [configs](#Configuration)), but not _always_
* make all config exposed through a `viper` registry
* leave developers a free-hand if they want to use all the knobs and features proposed by `cobra`

There a few great libraries around to build a CLI (see below, a few that I like). `cobra` stands out as most likely the richest and most flexible,
as CLIs are entirely built programmatically.
* it should be easier to interact with command line flags of various types
* simple, declarative registration and binding of flags to config
* allow CLI flags to override this config (12(factors))
* includes slices, maps and custom flag types -> deferred to `github.com/fredbi/gflag`

This is great, but after some time spent building CLIs again and again, it became cleat that dealing with CLI flags and configs came with
a lot of repeatitive boiler-plate patterns.
* it should be easier to declare defaults (for flags, for config)

I felt the need to take side and adopt a few opinions to build more elegant CLIs.
* it should be easier to inject external dependencies into the commands tree (config, logger, etc)

The goals for this lib are:
* to integrate a CLI easily with config files (see [configs](#Configuration))
* to make all config exposed through a `viper` registry
* to allow CLI flags to override this config
* to remove the boiler-plate code needed to register, then bind the flags to the config registry
* to work more easily with flags of various types, including slices or custom flag types -> defered to `github.com/fredbi/gflag`
* to remove the need for the typical `init()` to perform all this initialization
* to remove the need to use package-level variables
* to design with testability in mind
#### Code style goals

Non-goals:
* to use struct tags: we want to stick to the programmatic approach - there are other great libraries around following the struct tags approach
* to use codegen: we want our code to be readable, not generated
* adopt a functional style (some would say DSL-like), with builder functions for CLI components
* the tree-like structure of commands should appear visually and obviously in the source code
* remove the need for the typical `init()` to perform all this initialization
* remove the need to use package-level variables
* remove the boiler-plate code needed to register, then bind the flags to the config registry
* remove the cognitive burden of remembering all the `GetString()`, `GetBool()` etc methods: `go` now has generics for that
* favor the use of generics exposed by `github.com/fredbi/gflags`, but don't require it
* design with testability in mind: CLI's should be testable with reasonable code coverage

#### Non-goals

* don't use struct tags: we want to stick to the programmatic approach - there are other great libraries around following the struct tags approach
* don't use codegen: we want our code to be readable, not generated

### Example for CLI

Sample CLI-building code. This is an excerpt taken from [this testable example](cli/example_test.go):

```go
import (
"fmt"
Expand Down Expand Up @@ -102,67 +131,186 @@ func main() {

## Configuration

The `config` package proposes an opinionated approach to dealing with config files,
and exposes a few configuration loaders that know about environments and secrets.
The `config` package proposes an opinionated approach to dealing with config files on top of `github.com/spf13/viper`.

It exposes configuration loaders that know about the context
(e.g a deployment environment such as `dev`, `production`) and secrets.

Although developped primarily to serve a CLI, this package may be used independently.

### Goals

The goals for this package are as follows:
#### Desirable features

* load configuration files, using sensible defaults from the powerful `github.com/spf13/viper` package.
* merge configurations, overloading value for a specific environment
* deal with the specifics of merge secrets in config
* help with testing the programs that consume configurations
* leverages all the 12-factor app stuff from `viper`
* leave developers a free-hand if they want to use all the knobs and features proposed by `viper`
(e.g. dynamic watch, remote config, etc)
* defaults are configurable

#### Code-style goals

* less boiler plate to deal with `viper` configuration settings and merging

#### Non-goals

* Avoid too much of automagically resolving things

> As much as a like what's available for pythonists with [Dynaconf](https://www.dynaconf.com/),
> I found myself spending too much time reading their doc to understand their default settings
> and figure out whetheir to adopt the default or override.
>
> This is a pitfall that is very difficult to avoid, and only experience and feedback will tell.

* Don't want to support older config formats such as `.ini`, `.toml` etc

> While perfectly doable, I prefer at the moment to focus on having less things to describe and
> document. So I believe that YAML and JSON are good enough.
* At the moment, no particular goal is set to support secrets via APIs (e.g. Hashicorp's Vault, Azure Vault...)

> Let's wait for a bit.
> At the moment, I am assuming secrets are just plain files (e.g. Kubernetes secret)
* to load configuration files, using sensible defaults from the powerful `github.com/spf13/viper` package.
* to merge configurations, overloading value for a specific environment
* to deal with the specifics of secrets in config
* to help with testing the programs that consume such configuations
* to save most of the boiler plate need to deal with viper configuration settings and merging.

### Approach to configuration

The goal of this approach is to merge configuration files with environment specifics.
We want to:
1. retrieve a config organized as a hierarchy of settings, e.g. a YAML document
2. merge configuration files with environment-specific settings
3. merge configuration files with secrets, usually these are environment-specific
4. clearly isolate and merge default settings

In addition,

* we want the hierarchy to be agnostic to the environment context
* most of the time, we don't want env-specific sections to propagate to the app level
(e.g. in the style of `.ini` sections)

Applications will then be able to consume the settings from a viper configuration registry.
> In our code, we should never check for a dev or prod specific section of the configuration.
Applications are able to consume the settings from a single viper configuration registry.

Supported format: YAML, JSON

Supported file extensions: "yml", "yaml", "json"

Folder structure:
### Example: loading a config

```go
import (
"fmt"
"log"
"os"

"github.com/fredbi/go-cli/config"
)

func ExampleLoad() {
here := mustCwd()
os.Setenv("CONFIG_DIR", filepath.Join(here, "examples"))

// load and merge configuration files for environment "dev"
cfg, err := config.Load("dev", config.WithMute(true))
if err != nil {
err = fmt.Errorf("loading config: %w", err)

return
}

fmt.Println(cfg.AllSettings())
}

func mustCwd() string {
here, err := os.Getwd()
if err != nil {
err = fmt.Errorf("get current working dir: %w", err)
log.Fatal(err)
}

return here
}
```

See other [examples](.config/examples_test.go)

### Folders structure for configurations

By default we have:
```
# <- root configuration
{base path}/config.yaml
# <- environment-specifics
config.d/
# <- configuration to merge for environment
{environment}/config.yaml
# other environment-specifics ....
{...}/config.yaml
{base path}/config.yaml
# <- environment-specifics folder
config.d/
# <- extra configuration to merge
config.yaml
# <- possibly with a modified name: config.*.yaml
config.default.yaml
# <- configuration to merge for environment
{environment}/config.yaml
# other environment-specifics ....
{...}/config.yaml
```

When using default settings (these are configurable), the base path is defined by the `CONFIG_DIR` environment variable.
[Here is an example](./config/examples)

When using default settings for this module (these are configurable),
the base path is defined by the `CONFIG_DIR` environment variable.

Secret configurations:
```
{base path}/secrets.yaml
config.d/
# <- configuration to merge for environment
{environment}/secrets.yaml
config.d/
# <- secrets to merge
secrets.yaml
# <- configuration to merge for environment
{environment}/secrets.yaml
```

### Features

* sensible defaults for minimal boiler plate
* most defaults are configurable
* extensible
* the viper object may be watched dynamically
* can merge plain config with secrets
* helper method to easily inject configurations in test programs

### Typical configuration of a kubernetes deployment
### Typical configuration for a Kubernetes deployment

Typically, the configuration files are held in one or several `Configmap` resources, mounted by your deployed container.
Most likely, secret files will be mounted from `Secret` resources.

(... to be continued...)
Secret files can be mounted from `Secret` resources in the container, accessible as plain files.

Alternatively, Kubernetes may expose secrets as environment variables: `viper` takes care of loading them in the registry.

Normally, we don't want to expose secrets via CLI flags.

Example (e.g. volumes & container section of a k8s PodTemplateSpec):
```yaml
volumes:
- name: config
configMap: # <- expose config file from ConfigMap resource to the pod's containers
name: 'app-config'
- name: secret-config # <- expose secrets file from Secret as file resource to the pod's containers
secret:
secret_name: 'app-secret-config'

containers:
- name: app-container
...
env:
- name: CONFIG_DIR
value: '/etc/app'
- name: SECRET_URL
valueFrom:
secretKeyRef: # <- expose config value as an environment variable to the container
name: 'app-secret-url'
key: secretUrl

volumeMounts:
- mountPath: '/etc/app' # <- mount config file(s) as /etc/app/{key(s)} file(s)
name: config
- mountPath: '/etc/app/config.d'
```
### Side notes
(... to be continued...)
#### Dealing with secrets locally
TODO(fredbi)
3 changes: 3 additions & 0 deletions config/TODO.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# TODOs

* [ ] Support multiple mount points?
* [ ] Clearer view on resolving configs / secrets, order merges are carried out etc
* [ ] More examples
* [ ] Better testability for local secrets
* [ ] Support for viper remote configurations (e.g. consul, etc..)
* [ ] Write YAML of merged CLI
53 changes: 39 additions & 14 deletions config/examples_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,34 +9,59 @@ import (
"github.com/fredbi/go-cli/config"
)

func ExampleLoadWithSecrets() {
// loads a config, merge clear-text secrets, then save the result file.
func mustCwd() string {
here, err := os.Getwd()
if err != nil {
err = fmt.Errorf("get current working dir: %w", err)
log.Fatal(err)
}

var err error
defer func() {
if err != nil {
log.Fatal(err)
}
}()
return here
}

func mustTempDir() (string, func()) {
folder, err := os.MkdirTemp("", "")
if err != nil {
err = fmt.Errorf("creating temp dir: %w", err)

return
log.Fatal(err)
}

defer func() {
return folder, func() {
_ = os.RemoveAll(folder)
}()
}
}

here, err := os.Getwd()
func ExampleLoad() {
here := mustCwd()
os.Setenv("CONFIG_DIR", filepath.Join(here, "examples"))

// load and merge configuration files for environment "dev"
cfg, err := config.Load("dev", config.WithMute(true))
if err != nil {
err = fmt.Errorf("get current working dir: %w", err)
err = fmt.Errorf("loading config: %w", err)

Check failure on line 41 in config/examples_test.go

View workflow job for this annotation

GitHub Actions / lint 

ineffectual assignment to err (ineffassign)

return
}

fmt.Println(cfg.AllSettings())

// Output:
// map[app:map[threads:10 url:https://example.dev.co] log:map[level:info] metrics:map[enabled:true exporter:prometheus] trace:map[enabled:true exporter:jaeger]]
}

func ExampleLoadWithSecrets() {
// loads a config, merge clear-text secrets, then save the result file.
var err error
defer func() {
if err != nil {
log.Fatal(err)
}
}()

folder, clean := mustTempDir()
defer clean()
here := mustCwd()

os.Setenv("CONFIG_DIR", filepath.Join(here, "examples"))

// load and merge configuration files
Expand Down

0 comments on commit c2bc4d7

Please sign in to comment.