Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow configuration file to be specified by an environment variable #543

Closed
kevinjqiu opened this issue Jun 26, 2018 · 7 comments
Closed

Allow configuration file to be specified by an environment variable #543

kevinjqiu opened this issue Jun 26, 2018 · 7 comments

Comments

@kevinjqiu
Copy link

@kevinjqiu kevinjqiu commented Jun 26, 2018

Similar to how kubectl loads configuration files, if KUBECONFIG environment variable is set, it will load that config file as opposed to $HOME/.kube/config.

We have a few CLI scripts built on top of invoke, and we'd like the ability to specify config files during runtime.

e.g., suppose our CLI needs to talk to different servers for different environments,

inv -f dev.yaml <command>
inv -f prd.yaml <command>

we'd like to have the ability to do:

export INV_CONFIG=$(pwd)/dev.yaml
inv <command>

@bitprophet if you think this is a worthy feature, I wouldn't mind contributing a PR.

Thanks for pyinvoke!

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Jun 27, 2018

I'd like this to be possible, sure! Most env var use so far has focused on the "env vars as a late-binding config layer" and this definitely can't work via that avenue because the runtime conf file wants to load before the env var layer.

So it'll need to be "regular" env var lookup, sort of like the early use of INVOKE_DEBUG. (Which is then also, later, shoved into the config itself as an override of the top level debug setting – not something you need to worry about for this feature, I don't think.)

I was all set to give you a quick checklist including "write tests for this" but I went and looked and we're not testing INVOKE_DEBUG either. So I'm gonna just go try and stab this directly real quick. Famous last words.

bitprophet added a commit that referenced this issue Jun 28, 2018
bitprophet added a commit that referenced this issue Jun 28, 2018
Implementation is always ~1/10 of tests/docs...
@bitprophet
Copy link
Member

@bitprophet bitprophet commented Jun 28, 2018

All set, is now the INVOKE_RUNTIME_CONFIG env var, let me know how it looks if you want to download the master branch! Will go out in the 1.1 release otherwise. Kinda want to tackle at least one more real feature before I put that on PyPI. Hopefully soonish.

@bitprophet bitprophet closed this Jun 28, 2018
@kevinjqiu
Copy link
Author

@kevinjqiu kevinjqiu commented Jun 28, 2018

@bitprophet Thanks for the quick turnaround. One request: can we config the env var that invoke looks for? We customize invoke to build our CLI with a different name, and it'd be nice to have the environment variable reflect our CLI's name.

e.g., if our CLI is awesome, the env var we'd like awesome to look for is AWESOME_CONFIG.

Thanks!

@kevinjqiu
Copy link
Author

@kevinjqiu kevinjqiu commented Jun 28, 2018

But as you said, the INVOKE_RUNTIME_CONFIG is a "regular env var" unlike the late-binding env vars. I tried the tester program on master with the latest RUNTIME_CONFIG feature, setting TESTER_RUNTIME_CONFIG (after using a specialized Config class with prefix set to tester) didn't work:

$ tree
.
|____setup.py
|____tester
| |____tasks.py
| |____main.py
|____config.yaml
$ cat tester/main.py 
from invoke import Program, Collection, Config
from tester import tasks

class TesterConfig(Config):
    prefix = 'tester'


program = Program(version='0.1.0',
                  namespace=Collection.from_module(tasks),
                  config_class=TesterConfig)
$ cat tester/tasks.py 
from invoke import task


@task
def hello(ctx):
    print('Hello world: {}'.format(ctx.config.foo))
$ cat config.yaml 
foo: bar
$ tester -f config.yaml hello
Hello world: bar
$ export INVOKE_RUNTIME_CONFIG=$(pwd)/config.yaml
$ tester hello
Hello world: bar
$ unset INVOKE_RUNTIME_CONFIG
$ export TESTER_RUNTIME_CONFIG=$(pwd)/config.yaml
$ tester hello
Traceback (most recent call last):
  File "/home/kevin/src/invoke/invoke/config.py", line 112, in __getattr__
    return self._get(key)
  File "/home/kevin/src/invoke/invoke/config.py", line 177, in _get
    value = self._config[key]
KeyError: 'foo'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kevin/src/invoke/.env/bin/tester", line 11, in <module>
    load_entry_point('tester', 'console_scripts', 'tester')()
  File "/home/kevin/src/invoke/invoke/program.py", line 335, in run
    self.execute()
  File "/home/kevin/src/invoke/invoke/program.py", line 491, in execute
    executor.execute(*self.tasks)
  File "/home/kevin/src/invoke/invoke/executor.py", line 134, in execute
    result = call.task(*args, **call.kwargs)
  File "/home/kevin/src/invoke/invoke/tasks.py", line 128, in __call__
    result = self.body(*args, **kwargs)
  File "/home/kevin/src/invoke/tmp/tester/tasks.py", line 6, in hello
    print('Hello world: {}'.format(ctx.config.foo))
  File "/home/kevin/src/invoke/invoke/config.py", line 124, in __getattr__
    raise AttributeError(err)
AttributeError: No attribute or config key found for 'foo'

Valid keys: ['run', 'runners', 'sudo', 'tasks']

Valid real attributes: ['clear', 'clone', 'env_prefix', 'file_prefix', 'from_data', 'global_defaults', 'load_base_conf_files', 'load_collection', 'load_defaults', 'load_overrides', 'load_project', 'load_runtime', 'load_shell_env', 'load_system', 'load_user', 'merge', 'pop', 'popitem', 'prefix', 'set_project_location', 'set_runtime_path', 'setdefault', 'update']
@bitprophet
Copy link
Member

@bitprophet bitprophet commented Jul 5, 2018

Sorry, for some reason I'd interpreted what you said as a generic question instead of a specific followup – you're right, that new env var we just added is hardcoded, I'll make a new ticket for making it configurable. (Smells a little off -- think it may want tighter tie-in with how Program and Config interact re: env vars and prefixes -- but still something to put down as a nice-to-have.)

@bitprophet
Copy link
Member

@bitprophet bitprophet commented Jul 5, 2018

See #550 for that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.