Make GoogleCloudServiceAccountDictProfileMapping dataset profile argument optional

[oliverrmaa]

Context

Given the issue here and the discussion around it in slack, it follows that it would not be too far off for other profile mapping classes (like GCP) to have their schema (dataset in GCP terms) arguments made optional.

The configurations in dbt (profiles.yml, dbt_project.yml, model level configs etc.) are meant to provide the dataset. As such, me and my team's initial thoughts were to not provide any arguments here since usually we would rely on this being handled inside of dbt instead of it being hard coded at the DAG (orchestration) level.

Solution

I believe its possible modifying this line here and the associated code similarly to this issue here would be the right path.

Me and/or my colleague are happy to to attempt this modification if there are no objections to this proposed change. Please let us know, thanks

[dosubot]

🤖

That sounds like a great idea! Making the dataset profile argument optional could provide more flexibility for users. Your proposed solution seems like a step in the right direction. We encourage you and your colleague to go ahead and attempt the modification. Thank you for your willingness to contribute to the project!

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[oliverrmaa]

@dosu-bot I have opened a PR #839 to address this issue and it has passed all tests and is ready for review, how do I request a review?

[dosubot]

🤖

To request a review for your PR #839, you can follow these steps:

1. Go to your pull request page on GitHub.
2. On the right side of the page, you'll see a section titled "Reviewers".
3. Click on the gear icon next to "Reviewers".
4. From the dropdown, select the person or team you want to review your changes.

If you don't see the "Reviewers" section, it might be because you don't have the necessary permissions. In that case, you can leave a comment on the PR asking for a review from the relevant person or team.

Regarding the proposed change to make the dataset profile argument optional in GoogleCloudServiceAccountDictProfileMapping, it seems like a reasonable suggestion. The dataset is indeed usually handled inside of dbt and making it optional would provide more flexibility. However, please note that any changes to the codebase should be thoroughly tested to ensure they don't introduce any regressions or unexpected behavior.

As for the modification you mentioned, it would involve changing the required_fields list and the airflow_param_mapping dictionary in the GoogleCloudServiceAccountDictProfileMapping class. You would need to remove dataset from required_fields and handle the absence of dataset in airflow_param_mapping. This would likely involve checking if extra.dataset exists before trying to use it.

Please ensure that your changes are in line with the Cosmos Contributing Guide and that all tests pass before requesting a review. If you have any further questions or need more guidance, feel free to ask.

For more information, you can refer to the Cosmos codebase, specifically the GoogleCloudServiceAccountDictProfileMapping class, and the Cosmos Contributing Guide.

I hope this helps! If you have any other questions, don't hesitate to ask.

Sources

• docs/contributing.rst
• cosmos/profiles/bigquery/service_account_keyfile_dict.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[tatiana]

It seems the schema property is mandatory in profiles.yml when using BQ:

Having a dbt project with profiles.yml:

airflow_db:
  target: dev
  outputs:
    bq:
      type: bigquery
      method: service-account
      project: <project>
      #schema: <dataset>
      threads: 2
      keyfile:  /usr/local/airflow/dbt/jaffle_shop/key.json

And the following

models:
  jaffle_shop:
      +schema: dataset

Fails in both dbt 1.5 and 1.8:

Running dbt-core==1.5.0:

dbt ls --target bq --profile airflow_db
10:13:24  Running with dbt=1.5.0
10:13:32  Encountered an error:
Runtime Error
  Credentials in profile "airflow_db", target "bq" invalid: 'schema' is a required property

Running dbt-core==1.8.2:

dbt ls --target bq --profile airflow_db     
10:15:31  Running with dbt=1.8.2
10:15:40  Encountered an error:
Runtime Error
  Credentials in profile "airflow_db", target "bq" invalid: Runtime Error
    Must specify schema

I also attempted to do this while