Add additional BigQuery authentication method

[WilliamDee]

Context

Currently, MetricFlow only allows a user to pass in a path to the google JSON credential of a service account type. However, we want to expose another method of authentication. In this PR, it allows a user to authenticate BigQuery through a method called ADC (Application Default Credentials). This way a user can authenticate via their own end user credentials. More information about ADC here (https://google.aip.dev/auth/4110)

Changes

• Renamed the config key in BigQuery config from dwh_password to dwh_pass_to_creds for clarity purposes
• Added a dwh_project_id config key if a user wants to auth via ADC
• modify the bigquery engine creation flow to accept both methods of passing credentials

Usage

Current Service Account Auth (No changes)

dwh_dialect: bigquery  # Dialect (one of BigQuery, Snowflake, Redshift)
dwh_path_to_creds: '<path_to_service_account_creds.json>'  # Provide the path to the BigQuery credential file, ignore to use ADC auth
dwh_schema: 'transform_staging'
dwh_project_id: ''  # Provide the GCP Project ID, ignore if using service account credentials

Note that dwh_project_id can be ignored

Additional Auth Method

Can be easily done by running gcloud auth application-default login on the command line which would create a credential file linked to your specific user to the default location provided by Google. This will then be used automatically when the BQ engine gets created. Then in the config file you will have something like this,

dwh_dialect: bigquery  # Dialect (one of BigQuery, Snowflake, Redshift)
dwh_path_to_creds: ''  # Provide the path to the BigQuery credential file, ignore to use ADC auth
dwh_schema: 'transform_staging'
dwh_project_id: '<project_id>'  # Provide the GCP Project ID, ignore if using service account credentials

Note that dwh_path_to_creds can be ignored

Disclosure

The recommended way of providing production authentication should be ideally through a service account credential and not an end user credential

[WilliamDee]

An important thing to note here is by changing the config key name here (from dwh_password to dwh_path_to_creds) any BQ user that upgrades to whatever version this change gets pushed out to will cause errors as the code will look for dwh_path_to_creds but their existing config file has dwh_password. Users will need to either

1. delete the config file and rerun mf setup

or

2. rename dwh_password to dwh_path_to_creds manually

[tlento]

An important thing to note here is by changing the config key name here (from dwh_password to dwh_path_to_creds) any BQ user that upgrades to whatever version this change gets pushed out to will cause errors as the code will look for dwh_path_to_creds but their existing config file has dwh_password

Something just occurred to me here.

What kind of error message will these users see? Can we make sure the error message provides a concrete indication of what they should do to fix the issue? Like if they have BQ as the warehouse type, can we have the error message tell them that they need to fill out the dwh_path_to_creds (and only that key)?

This isn't blocking on merge but we should follow up with that improvement before release.

[WilliamDee]

An important thing to note here is by changing the config key name here (from dwh_password to dwh_path_to_creds) any BQ user that upgrades to whatever version this change gets pushed out to will cause errors as the code will look for dwh_path_to_creds but their existing config file has dwh_password

Something just occurred to me here.

What kind of error message will these users see? Can we make sure the error message provides a concrete indication of what they should do to fix the issue? Like if they have BQ as the warehouse type, can we have the error message tell them that they need to fill out the dwh_path_to_creds (and only that key)?

This isn't blocking on merge but we should follow up with that improvement before release.

Added an error message. So for this specific case of a user using the old config version, they'll see

ValueError: One of `dwh_path_to_creds` or `dwh_project_id` should be filled.