Warning about unencryptet connection passwords if fernet_key is passed via environment variable.

[amerkel2]

If fernet_key is not specified in airflow.cfg, but passed via the environment variable AIRFLOW__CORE__FERNET_KEY, the warning about unencrypted passwords appears although passwords are encrypted. The reason is that the check uses ConfigParsers has_key(), which does not consider the environment.

[r39132]

Hi!
What is AIRFLOW__CORE__FERNET_KEY? I don't see that being referenced anywhere in the code base? I'm only aware of setting the variable via the config file. What steps did you take to reproduce this?

I will mention that if you had set the fernet key in the config and then edited a connection, the password would have been stored in the DB in encrypted form. If you, at some later point, removed the fernet key from your config file and set the ENV VAR, you will likely see the warming message though the password is encrypted in the DB.

If I understand your question, I believe that you are asking for Airflow to support setting config in the file and/or via ENV Vars. Is that what you are requesting?

[amerkel2]

Hi,

it's already possible to set config via environment variables. Quoting the Airflow documentation (http://pythonhosted.org/airflow/installation.html#configuration):

You can also set options with environment variables by using this format:
$AIRFLOW__{SECTION}__{KEY} (note the double underscores).

Thus, to set the fernet_key option in the section core, I use AIRFLOW__CORE__FERNET_KEY, which works fine, cf. https://github.com/airbnb/airflow/blob/master/airflow/configuration.py#L374.

The issue is that the ConfigParserWithDefaults class does not overwrite the has_option method of ConfigParser like it does with get. Hence, alert_fernet_keys (https://github.com/airbnb/airflow/blob/master/airflow/www/views.py#L2096), which uses that method, knows nothing about the environment variables mechanism and emits the warning when the option is not set in the config file.

[r39132]

Thanks for the explanation. I see the support in configuration.py for reading environment variables. Per the comments, it looks like ENV VARs trump (have precedence) config file values. I'm not a fan of having config in 2 places. More often than not, it results in unexpected behavior.

Why not place config exclusively in the file?

@mistercrunch This feature of having config in 2 places seems like more of a timebomb than a necessary feature. Thoughts?

[mistercrunch]

https://github.com/airbnb/airflow/pull/948/files

[mistercrunch]

Well there's actyally a 3rd option now with support for executable. The env vars seemed reasonable and apparently is a standard way to do things on the cloud. The "executing commands to get the stdout so we don't have to store the password in plaintext" thing came from Yahoo and apparently is a standard way of doing things there. That one I was more uneasy about but let it through. With the precedence rules it shouldn't get to that anyways.

With a good prefix AIRFLOW__{section}__{key} there's no risk of collision.

[r39132]

If we support the option of picking up configuration from multiple sources, then we need the functionality to allow admins or users to inspect the configuration that is being used on the webserver in order to see which config value wins in light of conflicts. This will avoid people pulling out their hair.

Currently, ConfigurationView only exposes the config in the airflow.cfg file and does not expose values pulled in from other sources, hence it will give a misleading impression of what configuration is being used. In the specific case of ENV VARS, which trump all other values, ConfigurationView will show the wrong value - i.e. the value from the config file which was trumped by the ENV VAR.

On a separate note, I only run Airflow on 1 machine. If folks are running multiple webservers and schedulers, there is an added complication of airflow.cfg differing on multiple machines.

class ConfigurationView(wwwutils.SuperUserMixin, BaseView):
    @expose('/')
    def conf(self):
        from airflow import configuration
        raw = request.args.get('raw') == "true"
        title = "Airflow Configuration"
        subtitle = configuration.AIRFLOW_CONFIG
        if configuration.getboolean("webserver", "expose_config"):
            with open(configuration.AIRFLOW_CONFIG, 'r') as f:
                config = f.read()

For the first issue, we should be able to update ConfigurationView to carry out conflict resolution and show winners. In order to solve the problem of multiple machines with different configuration, I don't have a quick solution.

[amerkel2]

Hi guys,

thanks for the fix. Our use case is exactly that we run Airflow in the cloud and don't want to have the Fernet key stored in a file.

[r39132]

@amerkel2 Unless I hear otherwise by EOD, I will close this issue.