Values from environment variables

[Mange]

I suggest an option to Arg where one can specify a default value to be read from an environment variable. This would of course be visible in the --help, and any type conversions should still take place.

Here's how it would look like (simplified):

Arg::with_name("redis-url")
  .long("redis-url")
  .value_name("URL")
  .help("The URL to the Redis server to use.")
  // .default_from_env("REDIS_URL")
  .default_from_env_or("REDIS_URL", "redis://localhost:6379/0")
  .required(true);

// --redis-url=URL   The URL to the Redis server to use.
//                   Defaults to the value of the REDIS_URL environment variable, or "redis://localhost:6379/0".

// Without hardcoded default:
// --redis-url=URL   The URL to the Redis server to use.
//                   Defaults to the value of the REDIS_URL environment variable.

If we had a validator attached, it would be run on the value of the environment variable just like if the user provided the string (it's what they are doing, after all).

Note how I'm not advocating automatically determining environment variables to use, or loading any .env files or such. There's no filesystem logic here. It's just to let the user assign the name of an environment variable to read from if the argument isn't part of the argv.

Background

With the advent of 12-factor apps that are commonly configured using environment variables, there's a lot of patterns and support for setting environment variables outside of a process and then reading them inside the program.

I've always felt that command line options are more discoverable and more usable from workstations, so I usually want them as well. Having to hook environment variables up with the command-line argument parser means a lot of work.

Gains

By adding this feature, it'll be much easier to have apps that can be configured with both environment variables and from the command line with just a single source of truth and parsing. I don't need to validate values in two different places, and I can model required values and conflicting values in one place.

This would motivate people to add a list of environment variables to their --help output as well.

It'll now be trivial to load a .env before parsing with Clap and get the best of both worlds.

Alternatives

There's #712, which seems to be the same suggestion, but larger in that it'll automatically determine and format certain things. Maybe I'm misunderstanding that issue's description. I apologize in that case.

One can also manually deal with this by reading the values and assigning them as defaults at runtime. Help messages can also be manually enhanced with such references. This still complicates the argument definitions a lot, however.

---

I'm a Rust beginner, but I'd be willing to come up with a proof of concept PR if that would be welcome.

[kbknapp]

I love the detailed write up! You're correct, this relates to #712 (env vars) and also to #748 (config files). #712 is proposing allowing use env vars to act as an external argv where SOME_VAR="--option val -f --other" or the like.

I don't see any reason why your proposal shouldn't land, as it's a simple addition and something I've been wanting to add anyways. I do have a few questions first though.

This would of course be visible in the --help

How so? I could potentially see, [env: SOME_VAR] listed in the help string? But I'd worry about there becoming too many "additional details." So my initial inclination is to leave this up to the consumer and use something like, This arg does X, and the value may be optionally specified in SOME_VAR But if someone is planning on having many, most, or all of their options allow corresponding env vars, the [env: SOME_VAR] is obviously easier.

---

I don't think I'd want to add default_from_env as that could allow SOME_VAR to not be used, and nothing provided on the command line. I'd prefer the default_from_env_or which garuntees a value.

[Mange]

I love the detailed write up!

Oh, thank you. ❤️

This would of course be visible in the --help

How so?

I wrote an example of how it could look like in the issue description:

// --redis-url=URL   The URL to the Redis server to use.
//                   Defaults to the value of the REDIS_URL environment variable, or "redis://localhost:6379/0".

I'm not interested in bike shedding this, but your suggestion with just [env: SOME_VAR] would most likely be enough for almost everyone. I like it as well. :-)

I don't think I'd want to add default_from_env as that could allow SOME_VAR to not be used, and nothing provided on the command line.

Isn't that just like it is today, when the value isn't provided? (foo --bar=baz vs foo)
Then required would fail the parsing if set to true or just move along if set to false, right?

I'm not very experienced in this library yet, so I think I might've misunderstood the different options.
Maybe what I'm suggesting shouldn't be called default, but rather a fallback? The point of the _or variant in my example was for fields that should also have a default, but mixing default_value together with this env "fallback" might be the wrong way to go.

But I'd worry about there becoming too many "additional details." [in the --help]

Maybe this should be a different approach, then. Just like with visible_alias / alias, maybe this should be visible_env_fallback / env_fallback?

Suggestion 2:

Arg::with_name("redis-url")
  .long("redis-url")
  .value_name("URL")
  .help("The URL to the Redis server to use.")
  .env_fallback("REDIS_URL") // Read this env variable when not provided in ARGV
  .default_value("redis://localhost:6379/0") // Then use this default value if no env variable is set
  .required(true);

Arg::with_name("email")
  .long("email")
  .value_name("EMAIL")
  .help("The author's email. Skip if you want to stay anonymous.")
  .env_fallback("USER_EMAIL")
  .required(false); // Not required, no default value. If it's not set in ARGV or ENV, it's just not set.

// I'm not sure this can be more contrived now...
// This just adds an option to "opt-out" of the ENV reading of --email, so to speak.
Arg::with_name("no-email")
  .long("anonymous")
  .help("Force anonymous authorship.")
  .overrides_with("email");

[EDIT: I see now that I might've misunderstood required. Just setting default_value probably does what I thought this did as I'd always get a value when reading then. required seem to say that it must be explicitly given by the user. Is this correct?]

[bachp]

@Mange I like your suggestion 2 as it would allow to make a parameter required and make sure it is either set as a parameter or as an environment variable. I'm looking for something like this to pass the authentication token via env variable for my gitlab-mirror project.

[bluejekyll]

Is anyone working on this? I've implemented annoying work arounds for this enough that I'd be willing to put together a PR, my preference being for Suggestion 2

[Mange]

I, for one, is not working on it. I opened this issue when I didn't feel confident enough in Rust to even try to implement it.

I feel different about it now, and could also take a stab if required.

[bachp]

I'm currently not working on this but a PR would be welcome 😉

[bluejekyll]

I decided on from_env, but if people like the env_fallback terminology I can change it.

[kbknapp]

Some important comments from #1057

My biggest concern for testing is to ensure correct order (from highest to lowest):

• Args from command line
• ENV vars
• Default Values

Also a test to confirm something along the lines of:

$ MYVAR='some' program --option

Where --option accepts 0 or one value. In this case, matches.value_of("option") == None because the empty value was specified at the command line (same for --option= and --option "").

The only one I'm a little unsure about is:

$ MYVAR='some' program --option

Where --option accepts 0 or more values (i.e. multiples). Should the result be ["", "some"] for the values, or None? The same goes for $ MYVAR='some' program --option=other. Should it be ["other", "some"] or ["other"]?

I'm actually fine with either way, so long as it's documented which way it'll happen. I'm inclined to lean specified args (at the command line) override env vars entirely, so the examples above would result in None and ["other"] respectively. Otherwise there is no way to "override" ENV vars without manually resetting them $ MYVAR="" program --option which would be a pain.

The case for "merging" env vars and command line options would be more in line with allowing external argv's as in $ MYARGV="--option some" program --option other. But that's a whole other issue 😜 so let's not worry about it here.

[kbknapp]

I decided on from_env, but if people like the env_fallback terminology I can change it.

I made the appropriate comments in #1057 but I'll copy them here too, so everyone is tracking:

It's nitpicky [or rather bikeshedable], but I'd rather use a name such as simply env or env_var so as not to confuse anyone. Typically, from_* are constructors which create the Arg from something, in a similar manner to From traits.

[doivosevic]

How would this work with derive approach? I'm looking at the docs but there is nothing there regarding derive https://docs.rs/clap/2.32.0/clap/struct.Arg.html#method.env

[epage]

@doivosevic please use discussions for getting support when using the API.

btw you linked to the clap 2 documentation which only works with structopt. To use the built-in derive API that replaced structopt, be sure to look at the clap 3 documentation.

In the clap 3 derive reference it describes raw attributes which is what you want to read for how to use this. There has been a lot of confusion over this, so we have a discussion trying to find ways to improve the situation