Add support for configuration file settings

[atc0005]

While this tool will be heavily command-line driven, there are likely to be several good use cases for laying out all desired options via configuration file.

The specific format is yet to be determined.

Potential options:

• YAML
• JSON (not likely)
• TOML (instead of INI)
• flat, space-separated text file

[atc0005]

flags-first package by Peter Bourgon supports all of these configuration file formats and command-line and environment variables.

https://github.com/peterbourgon/ff

[atc0005]

integrii/flaggy#24 (comment)

This GitHub issue shows how to use Save() and Load() methods on a custom Config object to handle loading and saving a configuration file. The example then covers how to use the struct with the flaggy package to overwrite default configuration settings with user-specified flags (if provided).

If we're willing to use a JSON-formatted configuration file, we could have the app save the current settings chosen on command-line out to a configuration file, either with an automatic choice for save/load location or something custom provided by the user (maybe an enhancement to the first draft).

[atc0005]

As noted on #2 (comment):

While researching logging, I saw that the https://github.com/spf13/viper package handles environment variables, config files and command-line flags. Using this package could tick-off multiple TODO items for this project.

[atc0005]

Aside from YAML, what other config file format easily support specifying multiple source paths to crawl/process?

[atc0005]

Aside from YAML, what other config file format easily support specifying multiple source paths to crawl/process?

TOML seems like it would.

This appears to be the most active TOML library for Go:

https://github.com/pelletier/go-toml

This project's README (https://github.com/naoina/toml) gives an example of breaking array values over multiple lines, which I think would be a good fit for specifying multiple paths for processing:

# Line breaks are OK when inside arrays
hosts = [
  "alpha",
  "omega"
]

Presumably the line break support is a part of the TOML spec and would be supported by most/all Go library implementations for TOML.

[atc0005]

Note to self: test both libraries:

• https://github.com/pelletier/go-toml
• https://github.com/go-yaml/yaml

Hands-on experience with both should make it easier to decide.

[atc0005]

Note to self: test both libraries:

• https://github.com/pelletier/go-toml
• ...

Takeaways thus far for TOML:

• case-sensitive (I initially missed this)
• pelletier/go-toml supports default values via the default struct tag (e.g., if a setting is missing from the config file)

Attempting to use default values may be tricky since we're also supporting command-line flags and environment variables.

[atc0005]

Takeaways thus far for TOML:

• case-sensitive (I initially missed this)

YAML is also case-sensitive.

[atc0005]

Scratch notes:

• Separate config objects per config source

  • config object for flags
  • config object for config file
  • config object for ... ?

• Separate NewConfig() from flags

  • LoadConfigFlags() (or another name) could apply flag settings to a config object for later merging with one or more other config objects

[atc0005]

Create base config object, then copy it for use as a template with a new config source?

[atc0005]

Note to self:

Comparing against a default:"" struct tag is problematic when the user explicitly opts to use that value.

Additionally, If the config file says to use json logging format, but the struct default says to use text and the args package uses that to configure its struct instance, how to tell whether the default from the struct was used vs a command-line flag or environment variable?

[atc0005]

Does go-arg have logic that prevents using the default struct tag it the target struct already has a configured value?

go-toml does not, so that might be irrelevant?

[atc0005]

At this point I've given up on trying to find a way to reliably keep the default struct tags in-place. While useful to have it appear in the help output, it allows the alexflint/go-arg and pelletier/go-toml packages to believe that they are the sole user of that field. This results in those packages assuming (perhaps rightly so) that if you didn't explicitly set a value, then the default tag on a struct field should be used to set/replace the value held in that field.

Thus if we try to use a layered configuration object approach with the default tags in place we lose any non-default settings applied from the "layer" beneath the current configuration object (e.g., processing flags after having just processed a config file).

In short, I think the default struct tags have to go for now.

[atc0005]

Some other scratch notes from a brainstorming session (which probably didn't catch everything). As I write this I've found a few logic errors I still have to work out. Transcribing my existing notes anyway in case it helps me look at the problem from a different angle later.

---

• Move default values back to constructor
  • context: the default struct tags currently used by alexflint/go-arg and pelletier/go-toml if found

1. Process env files to load into env vars
2. Create base config object
3. Merge file config object
4. Merge flags/env config object

MergeConfig() could have simpler logic:

• if source struct field is non-zero, then overwrite dest struct field
  • I later attempted this by using a defaultConfig object for comparison instead of a non-zero value comparsion (gets rid of "magic" strings and numbers
  • If we apply config checks in order, this should work
• the winning field is non-zero source field, otherwise existing destination field
  • note: this didn't work in practice initially until I made sure that the destination and source config structs were both fully initialized with intended default values (bug otherwise)
    -If default config is zero value and incoming config is zero value, we will do nothing and leave destination struct field as-is. This should result in a usable config object
  • as of this writing the branch I'm working with has odd behavior, so there is still some unaccounted for logic bugs somewhere

Config layering order:

1. "default" config
2. process env vars files
3. config file object
4. flags/env vars config object (e.g., alexflint/go-arg)

[atc0005]

Still testing/working with my earlier approach. I'm starting to see some issues with the simple logic (e.g., copying slice values between structs), so I spent a short bit looking for community solutions to merging structs. Seems I'm far from the only one attempting to merge configuration settings across various structs, particularly from different configuration sources.

This package is used by quite a number of larger projects, including Kubernetes and Docker:

• https://github.com/imdario/mergo

here are some (unsorted) examples of using it to merge different structs:

• https://github.com/kubernetes/kubernetes/blob/70d453d32df254ef0c3420804c6487a2ba55a5e0/staging/src/k8s.io/client-go/tools/clientcmd/client_config.go#L187

• https://github.com/kubernetes/kubernetes/blob/70d453d32df254ef0c3420804c6487a2ba55a5e0/staging/src/k8s.io/client-go/tools/clientcmd/client_config.go#L434

• https://github.com/kubernetes/kubernetes/blob/70d453d32df254ef0c3420804c6487a2ba55a5e0/staging/src/k8s.io/client-go/tools/clientcmd/client_config.go#L277

• https://github.com/kubernetes/kubernetes/blob/70d453d32df254ef0c3420804c6487a2ba55a5e0/staging/src/k8s.io/client-go/tools/clientcmd/client_config.go#L430

• https://github.com/moby/moby/blob/c4cf72bad325f90ebf021d7cb52c262992a687c7/daemon/config/config.go#L374

[atc0005]

Still testing/working with my earlier approach. I'm starting to see some issues with the simple logic (e.g., copying slice values between structs), so I spent a short bit looking for community solutions to merging structs.

I'm not sure yet if the "stars aligned" or if I'm just fooling myself, but the current state of the associated feature branch appears to be working. I need to do further testing to make sure that config settings are being overridden as intended to establish the proper precedence.