Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use metadata to rewrite date params for dry run #2122

Merged
merged 2 commits into from
Jun 14, 2021

Conversation

relud
Copy link
Collaborator

@relud relud commented Jun 11, 2021

this handles query parameters as seen here

scheduling:
dag_name: bqetl_mozilla_vpn_site_metrics
date_partition_parameter: date

and here
scheduling:
dag_name: bqetl_mozilla_vpn
# destination is the whole table, not a single partition,
# so don't use date_partition_parameter
date_partition_parameter: null
parameters:
- "date:DATE:{{ds}}"

so that bqetl query schema update works for both those tables

@relud relud requested a review from scholtzan June 11, 2021 23:50
@relud relud force-pushed the dryrun-metadata-query-param branch from bcc7b81 to 1a59bef Compare June 11, 2021 23:50
@relud relud force-pushed the dryrun-metadata-query-param branch from 1a59bef to 59b0954 Compare June 11, 2021 23:52
@relud relud enabled auto-merge (squash) June 14, 2021 19:39
@relud relud merged commit b273a08 into main Jun 14, 2021
@relud relud deleted the dryrun-metadata-query-param branch June 14, 2021 19:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants