MWAA and Environment Variables in profiles.yml

[lukedappleton]

Hello! I found your package tonight and am currently trying to get this set up. After finding workarounds for the below by essentially using the /tmp directory on MWAA, I believe I am on my last issue before successful deploy.

modules-path
target_path
log-path

I am trying to get secrets read in through my profiles.yml file. I have tried entries like the below but to no avail.

host: "{{ env_var('AIRFLOW__DB_ENDPOINT') }}"
host: "{{ env_var('AIRFLOW_DB_ENDPOINT') }}"
host: "{{ env_var('DB_ENDPOINT') }}"
host: "{{ env_var('AIRFLOW__DB_ENDPOINT', 'not_set') }}"

The last option let me bypass the error temporarily before it failed when building the model. I know this may not be entirely specific to your package, but noticed that you have helped people out before with issues trying to get MWAA running.

https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-env-variables.html - I was using this somewhat but have not yet actually added anything under custom options. As you can see above, I tried adding AIRFLOW_ and AIRFLOW__ to the beginning but no luck.

Thought I would reach out and see if you have had luck with something similar. We just switched over to using AWS's secret manager in MWAA and its been working like a charm in our Python tools and it still uses Variable.get() (from Airflows library).

The typical error I get is:

Env var required but not provided: 'DB_ENDPOINT'

but have also gotten this:

Env var required but not provided: 'DB_ENDPOINT'. This can happen when calling a macro that does not exist. Check for typos and/or install package dependencies with "dbt deps".

Any help would be appreciated if you have the time!

[lukedappleton]

We ended up creating a plugin that set the os environment variables to the corresponding Variables.get and that worked.

[dnascimento]

@lukedappleton would you mind sharing the plugin? I’m solving the exact same problem

[lukedappleton]

@dnascimento It would just be something like this.

from airflow.plugins_manager import AirflowPlugin
from airflow.models import Variable
#Define the plugin class
class MyPlugins(AirflowPlugin):
    #Name your AirflowPlugin
    name = "my_plugin"
    os.environ.get(variable_name) = Variable.get(variable_name)

then just save it in your plugins folder and Airflow will initialize it automatically.

[lrahal]

Hey! I have the same issue. There's still no better way that creating a special plugin for this?
Thanks!