Dbarok/athena dbt utils

[dbarok]

This is a:

• bug fix PR with no breaking changes — please ensure the base branch is master
• new functionality — please ensure the base branch is the latest dev/ branch
• a breaking change — please ensure the base branch is the latest dev/ branch

Description & motivation

In our team we use dbt extensively hence we found it useful to have dbt-utils available also for Athena.

Checklist

• I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)

  • BigQuery
  • Postgres
  • Redshift
  • Snowflake
  • Athena

• I have "dispatched" any new macro(s) so non-core adapters can also use them (e.g. the star() source)

• I have updated the README.md (if applicable)

• I have added tests & descriptions to my models (and macros if applicable)

• I have added an entry to CHANGELOG.md

[dbarok]

Hi thank you very much for dbt-utils, I added these changes so that we can use this also on Athena. Doing this for the first time so I'll appreciate any feedback or comments. Cheers! :)

[jtcohen6]

@dbarok This looks awesome! I'm glad you were able to add Athena support with just a handful of macros.

We're trying to be really careful about the scope of this package, which has ballooned significantly over the past few years. As such, we've invested in functionality that enables adapter-specific macros to live in external "shim" packages. These packages can be installed alongside dbt-utils and, by setting a project-level var (before v0.19.2) or dispatch config (v0.19.2 and going forwards), users can tell dbt to look for implementations of dbt_utils macros in these "shim" packages, too.

You can check out some example "shim" packages that are already up & running:

• spark-utils
• tsql-utils
• materialize-dbt-utils

A hypothetical athena-utils package would be responsible for:

• Defining just the athena__ version of each macro
• Setting up CI to run integration tests against an Athena cluster

How does that sound to you?

[dbarok]

Great thanks for the explanation and the advice @jtcohen6 🙂 ! I will try to do it that way then.

[courentin]

Thank you @dbarok for your work! I implemented the suggested solution of @jtcohen6 in the repository https://github.com/lalalilo/athena-utils
Please note that there are no integration tests for now, but it's a first draft and any PRs are welcome!

Thanks again!

[jasnonaz]

Thanks for moving to the separate package!! Going to close this PR.