Read all config from the environment

[naved001]

There are some config options that are set via the CLI, some via environment variables and some that are hard-coded. We should read all config options from the environment so it's easy to make changes between different deployments of the code. I am open to other opinions.

[jimmysway]

@naved001 I'm working on this right now and I have everything inside env files but I cant help but feel like the massive config and env file is a little cumbersome perhaps. Is there ever a world where we split configuration by keeping endpoints and secrets in environment variables/secret stores and moving business policy (rates, SU definitions, GPU mappings, rounding rules) into versioned YAML files (config/environments/*.yaml, policy.yaml, su_definitions.yaml, mappings.yaml) validated by a schema or is that not necessary. I dunno how complex everything is going to get here but it just feels instinctually wrong to throw everything inside a massive env file

[naved001]

You are right, we kind of don't want to put every thing in a massive env file. At least, stuff related to the business logic comes directly from nerc-rates so we don't have to do any additional validation. The config I want to put in the config files is like various S3 bucket names, endpoints, secret names etc. Because I deploy this on openshift as a kubernetes cronjob I want to be able to look at the manifest and know immediately where things are going to (which bucket, what cluster etc).

Also, there are places where I have bucket names that default to some value and I now kind of hate it, because I the often have to look at the code to find out how it goes to a bucket etc.

and moving business policy (rates, SU definitions, GPU mappings, rounding rules) into versioned YAML files (config/environments/*.yaml, policy.yaml, su_definitions.yaml, mappings.yaml) validated by a schema or is that not necessary.

We get most of that from nerc-rates (rates, SU definitions) and all that is validated in that repo. The only thing we could move is GPU mappings to a file instead of being hardcoded - not an issue yet. Perhaps, when we add a new GPU type that's when we could move those to a file.

[jimmysway]

For the first 'chunk' of this issue. I've just done the infrastructure constants