Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add optional additional_query_params config to provider DAGs #3533

Closed
stacimc opened this issue Dec 15, 2023 · 0 comments · Fixed by #3648
Closed

Add optional additional_query_params config to provider DAGs #3533

stacimc opened this issue Dec 15, 2023 · 0 comments · Fixed by #3648
Labels
💻 aspect: code Concerns the software code in the repository 🌟 goal: addition Addition of new feature 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs

Comments

@stacimc
Copy link
Contributor

stacimc commented Dec 15, 2023

Problem

It would be nice to add a conf option to the provider DAGs called additional_query_params, which would be merged with the results of get_next_query_params for each batch.

Description

An example use case for this option would be to run a provider DAG but restrict the search to a single user by adding a user_id query param to all requests.

Additional context

This would be handy, for instance, when backfilling records for a single user, as we wanted to do in #3490.

@stacimc stacimc added 🟩 priority: low Low priority and doesn't need to be rushed 🌟 goal: addition Addition of new feature 💻 aspect: code Concerns the software code in the repository 🧱 stack: catalog Related to the catalog and Airflow DAGs labels Dec 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💻 aspect: code Concerns the software code in the repository 🌟 goal: addition Addition of new feature 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

1 participant