add ability to pass in all config values as environment variables. #1385

Merged
merged 4 commits into from Jan 8, 2017

Projects

None yet

5 participants

@hramezani
Contributor

add environment variables that starts with GUNICORN_ like GUNICORN_WORKERS as settings.
skip variables that does not start with GUNICORN_, and also skip variable that not in settings like GUNICORN_FOO

@tilgovi

Overall, I like the idea, but interested to hear others' thoughts.

gunicorn/config.py
@@ -69,6 +69,19 @@ def set(self, name, value):
raise AttributeError("No configuration setting for: %s" % name)
self.settings[name].set(value)
+ def parse_env_vars(self):
+ env_vars = {}
+ for k,v in self.env_orig.items():
@tilgovi
tilgovi Dec 20, 2016 Collaborator

Same nitpick here about spacing.

gunicorn/config.py
+ for k,v in self.env_orig.items():
+ if k.startswith('GUNICORN_'):
+ try:
+ name = k.split('GUNICORN_', 1)[1].lower()
@tilgovi
tilgovi Dec 20, 2016 Collaborator

I don't think this can ever fail if k.startswith('GUNICORN_').

gunicorn/config.py
+ name = k.split('GUNICORN_', 1)[1].lower()
+ except:
+ continue
+ if not name or name not in self.settings:
@tilgovi
tilgovi Dec 20, 2016 Collaborator

name not in self.settings should be sufficient, no?

gunicorn/app/base.py
@@ -154,6 +154,11 @@ def load_config(self):
if default_config is not None:
self.load_config_from_file(default_config)
+ # Load up environment configuration
+ env_vars = self.cfg.parse_env_vars()
+ for k,v in env_vars.items():
@tilgovi
tilgovi Dec 20, 2016 Collaborator

small linting nit here, space between ,v.

@hramezani
Contributor

@tilgovi thanks, i fix problems.

@berkerpeksag
Collaborator

I also like the idea, thanks! I haven't reviewed the code yet, but we should document all GUNICORN_ env variables before merging this.

tests/test_config.py
@@ -285,3 +285,25 @@ def test_always_use_configured_logger():
c.set('statsd_host', 'localhost:12345')
# still uses custom logger over statsd
assert c.logger_class == MyLogger
+
+def test_load_enviroment_variables_config():
+ os.environ["GUNICORN_WORKERS"] = "4"
@berkerpeksag
berkerpeksag Dec 22, 2016 Collaborator

We need to unset these environment variables. I don't know what's the best practice to do this in pytest though (I'd suggest writing a context manager if we were using unittest -- perhaps you need a fixture?)

tests/test_config.py
+ os.environ["GUNICORN_FOO"] = "BAR"
+ with AltArgs():
+ app = NoConfigApp()
+ try:
@berkerpeksag
berkerpeksag Dec 22, 2016 Collaborator

This can be written as:

with pytest.raises(AttributeError):
    app.cfg.too
@hramezani
Contributor
hramezani commented Dec 22, 2016 edited

@berkerpeksag Thanks,
this PR accept KNOWN_SETTINGS in this format GUNICORN_setting_name. known setting names are in Settings. for example GUNICORN_bind, GUNICORN_worker_class, ...
Is this a good idea that a write a paragraph in settings page and mention that we can change all of settings in this page by setting environment variables with abov format?

@benoitc
Owner
benoitc commented Dec 22, 2016

What is the use case for this change? Can you elaborate about it?

@hramezani
Contributor
hramezani commented Dec 23, 2016 edited

@benoitc i saw Ability to pass in all config values as environment variables in issue list and @tilgovi commented:

I don't see any reason not to support it. Pull requests welcome.

finally i decide to do this.

with this change we can change gunicorn settings with environment variables.

@tilgovi
Collaborator
tilgovi commented Dec 23, 2016

There are times, especially in these days of containerization, where it is much easier to change an environment variable than to change the command line invocation or a configuration file.

@tilgovi
Collaborator
tilgovi commented Dec 23, 2016

We can do this in a separate PR, but I'm wondering if there is any reason to clear these keys from the environment when we spawn workers. I'm just wondering if any settings might contain information that we should try to prevent from leaking to workers and possibly then into a response.

@hramezani
Contributor

@benoitc @tilgovi
Is there anything that I do?

@berkerpeksag
Collaborator

Is this a good idea that a write a paragraph in settings page and mention that we can change all of settings in this page by setting environment variables with abov format?

It would be nice to add something like "Environment variable: GUNICORN_FOO" (or just plain GUNICORN_FOO) to every setting. For example, let's take a look at bind: http://docs.gunicorn.org/en/stable/settings.html#bind With "-b ADDRESS, --bind ADDRESS", we know that we can pass -b or --bind to gunicorn. We can document the env variable in the same way:

bind

* -b ADDRESS, --bind ADDRESS (CLI)
* GUNICORN_BIND (env variable)
* ['127.0.0.1:8000'] (default value)
@hramezani
Contributor

@berkerpeksag thanks for reply, I add documents.

@berkerpeksag
Collaborator

Thanks, docs/source/settings.rst is automatically generated. I prefer modifying https://github.com/benoitc/gunicorn/blob/master/docs/gunicorn_ext.py#L39 instead of manually adding these names. You can then run make -C docs html to update settings.rst (note that make html will make some unrelated changes in settings.rst -- please discard them before committing your changes)

docs/source/settings.rst
@@ -17,6 +17,7 @@ config
~~~~~~
* ``-c CONFIG, --config CONFIG``
+* ``GUNICORN_CONFIG (env variable)``
@berkerpeksag
berkerpeksag Dec 26, 2016 Collaborator

"(env variable)" can be removed. I used it as an annotation in my example :)

@hramezani
Contributor
hramezani commented Dec 26, 2016 edited

@berkerpeksag Thanks, whta is your opinion about adding a property like cli with name env_var and type boolean to Setting class? this can be used for following purpose:

  1. add variables to config if this propery = True.
  2. disable this functionality for some setting.
  3. generate document based on this property.
@benoitc
Owner
benoitc commented Dec 27, 2016

I'm not sure i'm comfortable with this change, it introduces an extra level of complexity in the configuration, having to document the variables to use is a sign. What about instead having a simple GUNICORN_CMD_ARGS (or whatever its name) where you pass the command line arguments that could be parsed like usual? Ie:

GUNICORN_CMD_ARGS ="--bind=127.0.0.1 --workers=3" ...

So we wouldn't have to document each changes and could reuse the current code to handle it. Thoughts?

@hramezani
Contributor

@benoitc Thanks, I change PR as your previous comment.

@decal
decal commented Jan 6, 2017

Looks even better than the Apache 2.4 envvars support..

@benoitc benoitc self-requested a review Jan 8, 2017
@benoitc
benoitc approved these changes Jan 8, 2017 View changes
@berkerpeksag

The PR looks pretty good to me, thanks! Just left some trivial comments.

We also need to document the new env variable.

gunicorn/config.py
@@ -69,6 +69,12 @@ def set(self, name, value):
raise AttributeError("No configuration setting for: %s" % name)
self.settings[name].set(value)
+ def get_gunicorn_env_var(self):
@berkerpeksag
berkerpeksag Jan 8, 2017 Collaborator

I'm not a native English speaker, but perhaps we can rename it to something like get_cmd_args_from_env? get_gunicorn_env_var sounds too generic to me. Ping @tilgovi :)

gunicorn/config.py
@@ -69,6 +69,12 @@ def set(self, name, value):
raise AttributeError("No configuration setting for: %s" % name)
self.settings[name].set(value)
+ def get_gunicorn_env_var(self):
+ env_vars = []
@berkerpeksag
berkerpeksag Jan 8, 2017 Collaborator

Unless I'm missing something obvious, env_vars is not needed here:

if 'GUNICORN_CMD_ARGS' in self.env_orig:
    return self.env_orig['GUNICORN_CMD_ARGS'].split()
return []
gunicorn/config.py
+ def get_gunicorn_env_var(self):
+ env_vars = []
+ if 'GUNICORN_CMD_ARGS' in self.env_orig:
+ env_vars = self.env_orig['GUNICORN_CMD_ARGS'].split()
@berkerpeksag
berkerpeksag Jan 8, 2017 Collaborator

Did you consider using shlex.split() here?

gunicorn/app/base.py
+ env_vars = self.cfg.get_gunicorn_env_var()
+ if env_vars:
+ env_args = parser.parse_args(env_vars)
+ for k, v in env_args.__dict__.items():
@berkerpeksag
berkerpeksag Jan 8, 2017 Collaborator

Wouldn't vars(env_args).items() be better?

tests/test_config.py
+ app = NoConfigApp()
+ assert app.cfg.workers == 4
+
+def test_cli_overrides_enviroment_variables_module():
@berkerpeksag
berkerpeksag Jan 8, 2017 Collaborator

Tests are look great, thank you! :) Could you also add a test that passes an invalid config option (e.g os.environ["GUNICORN_CMD_ARGS"] = "--monty-python"?

@hramezani
Contributor

@berkerpeksag Thanks, i fix all your comments.

@berkerpeksag
Collaborator

Looks great, thanks! I think you forgot to address two of my previous comments:

  • We need to cleanup the env variables we set in tests. We can do this in the following way:

    def test_cli_overrides_enviroment_variables_module(monkeypatch):
        monkeypatch.setenv("GUNICORN_CMD_ARGS", "--workers=4")
        # test code
    
  • GUNICORN_CMD_ARGS needs to be documented. We can mention it in docs/source/run.rst and docs/gunicorn_ext.py. We can leave this to a separate PR if you want.

@@ -285,3 +285,21 @@ def test_always_use_configured_logger():
c.set('statsd_host', 'localhost:12345')
# still uses custom logger over statsd
assert c.logger_class == MyLogger
+
@berkerpeksag
berkerpeksag Jan 8, 2017 Collaborator

Style nit: Please add two empty lines after each tests.

tests/test_config.py
+ os.environ["GUNICORN_CMD_ARGS"] = "--foo=bar"
+ with AltArgs():
+ with pytest.raises(SystemExit):
+ app = NoConfigApp()
@berkerpeksag
berkerpeksag Jan 8, 2017 Collaborator

Style nit: No need to set NoConfigApp() to app. Just call NoConfigApp().

@hramezani
Contributor
hramezani commented Jan 8, 2017 edited

@berkerpeksag thanks!, comments fixed.
can you help me for documentation? is this ok to add a note or paragraph in Settings and Running Gunicorn page, and describe that we can use all command line config in env vars in bellow format?
GUNICORN_CMD_ARGS ="--bind=127.0.0.1 --workers=3" ...

@berkerpeksag berkerpeksag merged commit 3f9eace into benoitc:master Jan 8, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@berkerpeksag
Collaborator

Thanks for the PR, @hramezani! :)

is this ok to add a note or paragraph in Settings and Running Gunicorn page, and describe that we can use all command line config in env vars in bellow format?
GUNICORN_CMD_ARGS ="--bind=127.0.0.1 --workers=3"

Yes, that sounds good to me. Let's open another pull request for documentation.

@hramezani
Contributor

@berkerpeksag thanks very much for your help. i open another pull request for documentation soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment