Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure we can support cross-project queries in Airflow #217

Closed
jklukas opened this issue Jul 5, 2019 · 1 comment
Closed

Ensure we can support cross-project queries in Airflow #217

jklukas opened this issue Jul 5, 2019 · 1 comment

Comments

@jklukas
Copy link
Contributor

jklukas commented Jul 5, 2019

As discussed in https://bugzilla.mozilla.org/show_bug.cgi?id=1563742, we will be starting to migrate all derived tables into the shared-prod project, so there will be an interim period where we will have tables split between derived-datasets:telemetry and shared-prod:telemetry_raw. As we migrate, we will be providing views in derived-datasets:telemetry and shared-prod:telemetry to ensure a consistent and uninterrupted interface for users.

During the interim, we will have some queries in Airflow writing to derived-datasets and some to shared-prod, with perhaps needing to read from a different project than they're writing into. We should check that we understand how to express this in our Airflow dags and that our entrypoint script is able to support these scenarios.

@jklukas
Copy link
Contributor Author

jklukas commented Aug 5, 2019

We are now doing this and have sufficient support while we transition.

@jklukas jklukas closed this as completed Aug 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant