Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Port over "How to configure credentials using YAML or env vars" from RTD to Docusaurus #3032

Original file line number Diff line number Diff line change
@@ -1,5 +1,101 @@
---
title: How to configure credentials using a YAML file or environment variables
---
import Prerequisites from '../../connecting_to_your_data/components/prerequisites.jsx'

This article is a stub.
This guide will explain how to use a YAML file and/or environment variables to populate credentials (or any value) in your ``great_expectations.yml`` project config.

<Prerequisites>

- [Set up a working deployment of Great Expectations](../../../tutorials/getting-started/intro.md)

</Prerequisites>


Steps
------

1. Decide where you would like to save the desired credentials or config values - in a YAML file, environment variables, or a combination - then save the values. In most cases, we suggest using a config variables YAML file. YAML files make variables more visible, easily editable, and allow for modularization (e.g. one file for dev, another for prod).

:::note

- In the ``great_expectations.yml`` config file, environment variables take precedence over variables defined in a config variables YAML
- Environment variable substitution is supported in both the ``great_expectations.yml`` and config variables ``config_variables.yml`` config file.

:::

If using a YAML file, save desired credentials or config values to ``great_expectations/uncommitted/config_variables.yml`` or another YAML file of your choosing:

```yaml
# great_expectations/uncommitted/config_variables.yml

my_postgres_db_yaml_creds:
drivername: postgres
host: 127.0.0.778
port: '7987'
username: administrator
password: ${MY_DB_PW}
database: postgres
```

:::note

- If you wish to store values that include the dollar sign character ``$``, please escape them using a backslash ``\`` so substitution is not attempted. For example in the above example for postgres credentials you could set ``password: pa\$sword`` if your password is ``pa$sword``. Say that 5 times fast, and also please choose a more secure password!
- When you save values via the CLI, they are automatically escaped if they contain the ``$`` character.
- You can also have multiple substitutions for the same item, e.g. ``database_string: ${USER}:${PASSWORD}@${HOST}:${PORT}/${DATABASE}``

:::

If using environment variables, set values by entering ``export ENV_VAR_NAME=env_var_value`` in the terminal or adding the commands to your ``~/.bashrc`` file:

```bash
export POSTGRES_DRIVERNAME=postgres
export POSTGRES_HOST=localhost
export POSTGRES_PORT='5432'
export POSTGRES_USERNAME=postgres
export POSTGRES_PW=''
export POSTGRES_DB=postgres
export MY_DB_PW=password
```

2. If using a YAML file, set the ``config_variables_file_path`` key in your ``great_expectations.yml`` or leave the default.

```yaml
# great_expectations/great_expectations.yml

config_variables_file_path: uncommitted/config_variables.yml
```

3. Replace credentials or other values in your ``great_expectations.yml`` with ``${}``-wrapped variable names (i.e. ``${ENVIRONMENT_VARIABLE}`` or ``${YAML_KEY}``).

```yaml
# great_expectations/great_expectations.yml

datasources:
my_postgres_db:
class_name: SqlAlchemyDatasource
data_asset_type:
class_name: SqlAlchemyDataset
module_name: great_expectations.dataset
module_name: great_expectations.datasource
credentials: ${my_postgres_db_yaml_creds}
my_other_postgres_db:
class_name: SqlAlchemyDatasource
data_asset_type:
class_name: SqlAlchemyDataset
module_name: great_expectations.dataset
module_name: great_expectations.datasource
credentials:
drivername: ${POSTGRES_DRIVERNAME}
host: ${POSTGRES_HOST}
port: ${POSTGRES_PORT}
username: ${POSTGRES_USERNAME}
password: ${POSTGRES_PW}
database: ${POSTGRES_DB}
```


Additional Notes
--------------------

- The default ``config_variables.yml`` file located at ``great_expectations/uncommitted/config_variables.yml`` applies to deployments created using ``great_expectations init``.