Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding the ability to override scheduling parameters on backfill #5364

Closed
braunreyes opened this issue Apr 11, 2024 · 0 comments · Fixed by #5540
Closed

Adding the ability to override scheduling parameters on backfill #5364

braunreyes opened this issue Apr 11, 2024 · 0 comments · Fixed by #5540
Labels
enhancement New feature or request

Comments

@braunreyes
Copy link
Contributor

braunreyes commented Apr 11, 2024

Currently the query backfill command only supports {{ds}} as a submission_date:DATE: scheduling parameter. However, it is possible for folks to add more complex logic for the normal bqetl runs.

For example: https://github.com/mozilla/private-bigquery-etl/blob/56190e888f051c7190241767d0b7519665bc70c9/sql/moz-fx-data-shared-prod/ads_derived/tiles_addressable_inventory_hourly_v1/metadata.yaml#L22

We need to backfill this table for a fill range of days, which means we do not need this custom logic for submission date. Given the way it is currently written, it seems as though providing the ability to completely override the list of parameters at backfill runtime is the best solution.

If the new flag is used, then scheduling parameters will be completely overridden. This will not be a 'merge' of parameters.

┆Issue is synchronized with this Jira Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
1 participant