Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

1032 move gcc puller tables to airflow variable #1035

Merged
merged 7 commits into from
Aug 23, 2024
Merged

1032 move gcc puller tables to airflow variable #1035

merged 7 commits into from
Aug 23, 2024

Conversation

gabrielwol
Copy link
Collaborator

What this pull request accomplishes:

  • move GCC layers and Primary key dictionaries to Airflow variables
  • Dynamic DAG generation for bigdata/ptc GCC layers using Airflow variable which stores credentials, deployment
  • added mat view refreshes to Airflow variable: easier addition of new aggregations!

Issue(s) this solves:

What, in particular, needs to reviewed:

  • I couldn't think of an obvious way to make mat view aggregations their own Airflow tasks in a dynamic way so I just added them to the pull_layers task. Thoughts?

What needs to be done by a sysadmin after this PR is merged

  • change data_scripts link in Morbius

@gabrielwol gabrielwol requested a review from chmnata August 9, 2024 17:56
@gabrielwol gabrielwol self-assigned this Aug 9, 2024
@gabrielwol gabrielwol linked an issue Aug 9, 2024 that may be closed by this pull request
Move gcc puller tables to Airflow variable #1032
Closed
chmnata
chmnata requested changes Aug 21, 2024
View reviewed changes
Copy link
Collaborator

@chmnata chmnata left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make the mapped task's name as the layer's name? for refreshing the materialized view, I'm thinking we could use a trigger function on the table instead 🤔

chmnata
chmnata reviewed Aug 22, 2024
View reviewed changes
dags/gcc_layers_pull.py Outdated
agg_sql = layer[1].get("agg")
if agg_sql is not None:
with conn.cursor() as cur:
print(agg_sql)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this for logging purpose?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I'll remove this, it was for testing (instead of execute).

chmnata
chmnata approved these changes Aug 22, 2024
View reviewed changes
Copy link
Collaborator

@chmnata chmnata left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for reminding me that morbius's airflow version doesn't have mapped tasks naming yet :meow_headache: This looks good! refreshing mat view as a tasks as good as well.

@gabrielwol gabrielwol merged commit 39e4258 into master Aug 23, 2024
3 of 4 checks passed
@gabrielwol gabrielwol deleted the 1032-move-gcc-puller-tables-to-airflow-variable branch August 23, 2024 13:27
@gabrielwol
Copy link
Collaborator Author

New issue to track the mapped task names: #1043

gabrielwol added a commit that referenced this pull request Sep 10, 2024
@gabrielwol
#1043 add map_index_template, fix name->item bug from #1035
677f5bd
gabrielwol added a commit that referenced this pull request Oct 7, 2024
@gabrielwol
#1043 add map_index_template, fix name->item bug from #1035
df2dc36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chmnata chmnata chmnata approved these changes

Assignees

@gabrielwol gabrielwol

Labels
Airflow enhancement gis
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Move gcc puller tables to Airflow variable
2 participants
@gabrielwol @chmnata