Improve config - CLI setup, Toml config, .envs

[joepio]

The current configuration setup for the server is not ideal:

• Lot of boilerplate code
• Both env variables and ~/.config/atomic/config.toml are used
• Running from CLI does not really instruct user to setup basic, required fields

Strategy

• Upgrade to clap v3 (currently in beta)
• Use a StructOpt based on the config.rs file
• Set defaults and env names using decorators

[joepio]

I'm not really content with how the server is configured. There are some .env variables, but we also have a config.toml for user-specific settings which is used by the CLI... and we can pass arguments to atomic-server using clap.

It's all a bit much. At the very least, I want a very clear strategy on when to use what.

• Subcommands and arguments in the atomic-server cli are used for one-off actions, such as 'rebuild the index' or 'create a new invite'. If you want to change the configuration, go to...
• Environment variables are used for most server configuration. The only exception of which is...
• config.toml are only for client settings. These can also be relevant to atomic-cli (client).

[jonassmedegaard]

At #187 was brought up an idea:

I'm thinking maybe it would be easiest if there was a separate command for creating the .env in your current directory.

I think environment variables should be treated as ephemeral - not bet dumped to a file and not be slurped in from a file.
Maybe some container tools bootstrap their environments through freezing/thawing via files, but this is not a container tool.

[jonassmedegaard]

...to quote from an inspirational text:

Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.

[joepio]

I think you're right - using a .env as a configuration file is not the best approach.

I'm often torn between using the config.toml and .env. I like .toml files better for multiple reasons, but currently use the .env for more config uptions precisely because envs can be overwritten by some runtime / coordinating piece of software. To make things even worse, I also have some flags for the server.

I think this is how I currently decide whether to put it in env, flag or config.toml:

• If it's something that should be ran once / occasionally, use a flag
• If it's shared between various apps (e.g. atomic-cli, which also wants to use the default agent and its private key), use config.toml
• If it is something that devops people will likely set, put it in .env.

But it still feels... wrong.

So ideally, we don't read .env files. But I also don't want to have config options in both env vars and config.toml. What is the best way forward?

[jonassmedegaard]

To me it would be ideal if atomic-server...

• worked "out of the box" - with no options passed and no config files available and no special environment variables defined.
  • e.g. use ports 80 and 443 as default only when executed as root - use 8080 and 8443 as non-root, and if any of the default ports are occupied then emit a warning and switch all ports to use random high number instead (but only for by default - if any port is set in option/env/configfile then it is a hard fail if it is unavailable)
• could be operated solely using command-line options (as that is easiest to do interactively)
• could be operated solely using environment options (as that is easiest for container use - 12factor style)
• any configfile setting was overridable in either env or as option
• any env setting was overridable as option

Essentially I would use config files only as an alternative when I get tired of writing same option again and again on the command-like.

[joepio]

I hear you, thanks for the clear feedback. So better defaults for ports (the others should work out of the box), and if I understand you correctly you want to allow all types of setting configs (flag, env, file..), so users can decide how they want to set any attribute. I should make sure that doing this does not introduce a ton of boilerplate code, so I should find a crate to help with this.

Maybe the Viperius crate can.

[jonassmedegaard]

A setting shared across tools should use a broader namespace. E.g. env var ATOMIC_PORT and configfile setting atomic_port (or maybe port in default section in TOML?) and command-line option --port - but ATOMIC_SERVER_FOO for something specific to the server.

[jonassmedegaard]

Viperius seems unpopular and not well maintained and biased towards a certain Go style of handling things. But I have only looked at it very quickly.

[jonassmedegaard]

My impression is that clap is the sensible, popular, common library.

Seems you want the optional env feature of clap.

Also, please consider use clap_generate - TAB-completion in the shell is cool!

[jonassmedegaard]

Maybe a logic like this:

1. Define internal defaults of all options.
2. Look for config file (e.g. using dirs to support XDG lookup logic)
3. If a config file is found then use that as "external defaults": Override default settings for any settings from that config file
4. parse command-line options and environment variables using clap - i.e. option takes precendence over env

[joepio]

Seems you want the optional env feature of clap.

That's great, thanks! That's a very sensible approach.

Also, please consider use clap_generate - TAB-completion in the shell is cool!

I'll definitely look into that, deserves it's own issue.