Include DBT DRY RUN as a utilities plugin #1164

SBurwash · 2023-02-12T16:52:09Z

All Singer definitions are stored in /_data/taps/ or /_data/targets. The minimal requirement for adding a tap or target will match the following format:

commands:
  run:
    args:
      args:
        --target ${DBT_DRY_RUN_TARGET}
        --verbose
      definition: |
        Allows you to run the plugin. Requires that you specify a profile target.
        Note - Requires you run `dbt-bigquery compile` before executing.
      executable: dbt_dry_run_invoker

definition: |
  dbt-dry-run is a meltano plugin based on the awesome tool created by the team at AutoTraderUk.
  It allows dbt-bigquery users to validate if their models will run (without encuring costs).
  It leverages bigquery's [dry-run](https://cloud.google.com/bigquery/docs/dry-run-queries) feature.

  **A note**
  This plugin is at it's most powerful (in our opinion) in CI (we use github actions). It allows us to ensure that our changes will not break production.
domain_url: https://github.com/autotraderuk/dbt-dry-run
meltano_edk: true
executable: dbt-dry-run
keywords:
- dbt
- bigquery
- testing
- continuous-integrations
label: dbt-dry-run
logo_url: /assets/logos/utilities/dbt-dry-run.png
maintenance_status: active
name: dbt-dry-dryn
namespace: dbt-dry-run
next_steps: |
  To run this plugin for the first time, simply specify you DBT_BIGQUERY_PROFILES_DIR, DBT_BIGQUERY_PROJECT_DIR and DBT_DRY_RUN_TARGET.
  You will also need to compile your dbt models using the command `dbt-bigquery compile`.

  Once this is done, simply run `dbt-dry-run:run` and watch the magic happen!
pip_url: git+https://github.com/potloc/dbt-dry-run-ext
edk_repo: https://github.com/potloc/dbt-dry-run-ext
repo: https://github.com/autotraderuk/dbt-dry-run
settings:
  - name: project-dir
    kind: string
    value: ${MELTANO_PROJECT_ROOT}/transform/
    env: DBT_PROJECT_DIR
    description: Location of the directory hosting your `dbt_project.yml` file.
  - name: profiles-dir
    kind: string
    value: ${MELTANO_PROJECT_ROOT}/transform/profiles/bigquery/
    env: DBT_PROFILES_DIR
    description: Location of the directory hosting your `profiles.yml` file.
  - name: target
    kind: string
    description: Target profile in DBT.
variant: potloc

Checklist

Add/update the file in the appropriate folder (/taps or /targets). The name of the file should match the name of the tap. If there is already one, add a descriptor to the name such as -search.
Add/update the PNG logo image in /assets/logos/<taps or targets>. The image name must match the YAML file name.
Tag @tayloramurphy or @pnadolny13 to flag it for review. Or post to the #hub channel on Meltano slack.

Reviewer Checklist

Validate file against JSON Schema. https://www.jsonschema.net/home is an option.
Build website locally and validate everything works.

netlify · 2023-02-12T16:52:12Z

👷 Deploy request for meltano-hub pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	`f4649ec`

aaronsteers · 2023-02-13T16:24:06Z

Because this plugin and the underlying library is currently BigQuery only, I believe, what about using a name like dbt-bigquery-dryrun as a natural complement to dbt-bigquery? (At least on the hub, I mean. The underlying packages and repos can stay as they are.)

pnadolny13 · 2023-02-13T17:51:58Z

@SBurwash thanks for the PR! To fix the CI tests you'll need to remove entity_url and meltano_edk. The string meltano_edk can go in the keywords list to indicate that its EDK based. Also if you run the linter command in https://github.com/meltano/hub/blob/main/CONTRIBUTING.md#linters you it will attempt to fix your yaml to match our rules.

tayloramurphy · 2023-02-13T18:13:20Z

@SBurwash in addition to the other suggestions, what do you think of having the image be the dbt logo with the words "Dry Run" over it? Or even the BQ logo as well? I don't think just the checkmark would look good on the Hub currently.

SBurwash · 2023-02-13T19:34:39Z

@SBurwash in addition to the other suggestions, what do you think of having the image be the dbt logo with the words "Dry Run" over it? Or even the BQ logo as well? I don't think just the checkmark would look good on the Hub currently.

@tayloramurphy I'm super happy with that! I didn't like my logo to begin with, I just have no talent at designing logos

aaronsteers · 2023-02-13T22:02:42Z

_data/meltano/utilities/dbt-dry-run/dbt-dry-run.yml

+- testing
+- continuous-integrations
+label: dbt-dry-run
+logo_url: /assets/logos/utilities/dbt-dry-run.png


@SBurwash - Following up from @tayloramurphy's comment ... if it's easier than adding a new logo, you are also welcome to reuse this existing dbt logo if you'd like - or anything else in that same directory.

hub/_data/meltano/utilities/dbt-bigquery/dbt-labs.yml

Line 61 in 02a2c1f

logo_url: /assets/logos/utilities/dbt.png

This is consistent (today at least) with all the dbt flavors using the same logo:

@SBurwash I mocked this up - what do you think?

Awesome, thanks @tayloramurphy

@SBurwash np! If you could just add this to the utility folder and update the yaml to reference it please 😄

😍 PRETTY! 😄

SBurwash · 2023-02-14T03:00:21Z

@pnadolny13 I'm having some issues with my python version for the linter. Is this new version acceptable? I'm not sure what's bad order and what's improper values in the linter tests 😅

_data/meltano/utilities/dbt-dry-run/dbt-dry-run.yml

Co-authored-by: Pat Nadolny <patnadolny@gmail.com>

pnadolny13 · 2023-02-15T20:54:41Z

@pnadolny13 I'm having some issues with my python version for the linter. Is this new version acceptable? I'm not sure what's bad order and what's improper values in the linter tests 😅

@SBurwash we have an open issue around either disabling or making the linting step easier. I'd say once you have your new logo updated we can force merge this and I'll update the linting separately.

@tayloramurphy does that work?

tayloramurphy · 2023-02-15T20:58:16Z

@pnadolny13 works for me!

aaronsteers · 2023-02-15T22:00:17Z

Tangental, but sharing here because it may be relevant to folks on this thread:

I've logged a discussion in dbt-core around dry-run features in general - as discussed today in office hours. Feel free to watch/upvote/comment if interesting:

Schema-only dry-run feature - similar to a universal override of `materialize: view` or `LIMIT 0` dbt-labs/dbt-core#6989
Proposal for dry-run feature in dbt-core autotraderuk/dbt-dry-run#33

aaronsteers · 2023-02-15T22:05:22Z

_data/meltano/utilities/dbt-dry-run/dbt-dry-run.yml

+label: dbt-dry-run
+logo_url: /assets/logos/utilities/dbt.png
+maintenance_status: active
+name: dbt-dry-dryn
+namespace: dbt-dry-run


Suggested change

label: dbt-dry-run

logo_url: /assets/logos/utilities/dbt.png

maintenance_status: active

name: dbt-dry-dryn

namespace: dbt-dry-run

label: dbt-bigquery-dry-run

logo_url: /assets/logos/utilities/dbt.png

maintenance_status: active

name: dbt-bigquery-dry-dryn

namespace: dbt_bigquery_dry_run

As discussed in office hours, proposing we add bigquery- into the name where it will be presented in the Hub. This doesn't require changing the underlying repo but we should rename the file and path to match something like .../utilities/dbt-bigquery-dry-run/{variant_name}.yml where variant name might be autotraderuk or something else...

@aaronsteers I wonder what you think about updating the label to something like dbt-dry-run (BigQuery Only) and leaving the name and everything else the same as the repo/package name. It always feels a little funny to me to rename packages, although I know we do it for pipelinewise and airbyte so there is precedence for it.

@pnadolny13 re:

@aaronsteers I wonder what you think about updating the label to something like dbt-dry-run (BigQuery Only) and leaving the name and everything else the same as the repo/package name.

That could work. Do I remember correctly that label is used in search results and as the page title?

Presumably then, if a dbt-dry-run were created for Snowflake, a key difference would be in whether names of future variants collide when adding side-by-side to Meltano. But this is just speculation since we don't know if/when non-BigQuery implementations will come about... 🤔

But anyway, yeah, I think I'm fine with adding ' (BigQuery)' or ' (BigQuery Only)' to the label if that accomplishes what we need to regarding discoverability and expectation-setting for new users.

pnadolny13 · 2023-02-15T22:25:43Z

@SBurwash I almost forgot - you should use the repo maintainer as the variant name and set it in the default_variants.yml. Like dbt-dry-run: autotraderuk. Like AJ mentioned your yaml definition should also be labeled after this variant name .../utilities/dbt-dry-run/autotraderuk.yml.

SBurwash · 2023-02-16T02:45:49Z

@pnadolny13 @aaronsteers thanks for the feedback! All requested changes should have been applied, please let me know if you would like me to do anything else!

aaronsteers

🚀

_data/meltano/utilities/dbt-dry-run/autotraderuk.yml

pnadolny13

@SBurwash I fixed one quick typo. Looks good to me!

SBurwash · 2023-02-16T20:10:03Z

🚀

Created dbt dry run hub request

44ed7df

SBurwash requested review from sbalnojan, tayloramurphy, pnadolny13, rwfeather and cjohnhanson as code owners February 12, 2023 16:52

Added edk status + entity url

b45561b

aaronsteers reviewed Feb 13, 2023

View reviewed changes

aaronsteers mentioned this pull request Feb 13, 2023

Yaml key ordering rules in CI are pretty aggressive and not conducive to readability #1167

Open

SBurwash added 2 commits February 13, 2023 21:40

Removed offending urls and changed logo

95e38f5

Moved kind string

41cf12f

pnadolny13 reviewed Feb 14, 2023

View reviewed changes

_data/meltano/utilities/dbt-dry-run/dbt-dry-run.yml Outdated Show resolved Hide resolved

pnadolny13 reviewed Feb 14, 2023

View reviewed changes

_data/meltano/utilities/dbt-dry-run/dbt-dry-run.yml Outdated Show resolved Hide resolved

SBurwash and others added 2 commits February 14, 2023 13:22

Update _data/meltano/utilities/dbt-dry-run/dbt-dry-run.yml

36a2589

Co-authored-by: Pat Nadolny <patnadolny@gmail.com>

Update _data/meltano/utilities/dbt-dry-run/dbt-dry-run.yml

8a4f132

Co-authored-by: Pat Nadolny <patnadolny@gmail.com>

aaronsteers mentioned this pull request Feb 15, 2023

Proposal for dry-run feature in dbt-core autotraderuk/dbt-dry-run#33

Closed

Merge branch 'main' into main

8135d9b

aaronsteers suggested changes Feb 15, 2023

View reviewed changes

SBurwash added 2 commits February 15, 2023 21:41

Added changes

5c193f8

Merge remote-tracking branch 'refs/remotes/origin/main'

e43d1a0

Removed change for superset

7be689c

SBurwash requested review from pnadolny13 and aaronsteers and removed request for tayloramurphy, rwfeather, cjohnhanson, sbalnojan and pnadolny13 February 16, 2023 02:44

aaronsteers approved these changes Feb 16, 2023

View reviewed changes

pnadolny13 reviewed Feb 16, 2023

View reviewed changes

_data/meltano/utilities/dbt-dry-run/autotraderuk.yml Outdated Show resolved Hide resolved

Update _data/meltano/utilities/dbt-dry-run/autotraderuk.yml

f4649ec

pnadolny13 approved these changes Feb 16, 2023

View reviewed changes

tayloramurphy merged commit da5f240 into meltano:main Feb 16, 2023

This was referenced Feb 16, 2023

reformat dbt-dry-run #1173

Merged

add missing maintainer #1175

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Include DBT DRY RUN as a utilities plugin #1164

Include DBT DRY RUN as a utilities plugin #1164

SBurwash commented Feb 12, 2023 •

edited

Loading

netlify bot commented Feb 12, 2023 •

edited

Loading

aaronsteers commented Feb 13, 2023 •

edited

Loading

pnadolny13 commented Feb 13, 2023

tayloramurphy commented Feb 13, 2023

SBurwash commented Feb 13, 2023

aaronsteers Feb 13, 2023 •

edited

Loading

tayloramurphy Feb 14, 2023

SBurwash Feb 14, 2023

tayloramurphy Feb 14, 2023

aaronsteers Feb 16, 2023

SBurwash commented Feb 14, 2023

pnadolny13 commented Feb 15, 2023

tayloramurphy commented Feb 15, 2023

aaronsteers commented Feb 15, 2023

aaronsteers Feb 15, 2023

pnadolny13 Feb 15, 2023

aaronsteers Feb 16, 2023 •

edited

Loading

pnadolny13 commented Feb 15, 2023

SBurwash commented Feb 16, 2023

aaronsteers left a comment

pnadolny13 left a comment

SBurwash commented Feb 16, 2023

Include DBT DRY RUN as a utilities plugin #1164

Include DBT DRY RUN as a utilities plugin #1164

Conversation

SBurwash commented Feb 12, 2023 • edited Loading

Checklist

Reviewer Checklist

netlify bot commented Feb 12, 2023 • edited Loading

👷 Deploy request for meltano-hub pending review.

aaronsteers commented Feb 13, 2023 • edited Loading

pnadolny13 commented Feb 13, 2023

tayloramurphy commented Feb 13, 2023

SBurwash commented Feb 13, 2023

aaronsteers Feb 13, 2023 • edited Loading

Choose a reason for hiding this comment

tayloramurphy Feb 14, 2023

Choose a reason for hiding this comment

SBurwash Feb 14, 2023

Choose a reason for hiding this comment

tayloramurphy Feb 14, 2023

Choose a reason for hiding this comment

aaronsteers Feb 16, 2023

Choose a reason for hiding this comment

SBurwash commented Feb 14, 2023

pnadolny13 commented Feb 15, 2023

tayloramurphy commented Feb 15, 2023

aaronsteers commented Feb 15, 2023

aaronsteers Feb 15, 2023

Choose a reason for hiding this comment

pnadolny13 Feb 15, 2023

Choose a reason for hiding this comment

aaronsteers Feb 16, 2023 • edited Loading

Choose a reason for hiding this comment

pnadolny13 commented Feb 15, 2023

SBurwash commented Feb 16, 2023

aaronsteers left a comment

Choose a reason for hiding this comment

pnadolny13 left a comment

Choose a reason for hiding this comment

SBurwash commented Feb 16, 2023

SBurwash commented Feb 12, 2023 •

edited

Loading

netlify bot commented Feb 12, 2023 •

edited

Loading

aaronsteers commented Feb 13, 2023 •

edited

Loading

aaronsteers Feb 13, 2023 •

edited

Loading

aaronsteers Feb 16, 2023 •

edited

Loading