-
-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Airflow operator #215
Airflow operator #215
Conversation
0710b0e
to
36b51b5
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a few minor changes that I think can be cleaned up with a future PR. Thank you for this work! We've been meaning to clean this stuff up for a long time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lprzychodzien - why are these dag tasks here? Do we need to remove this file since you switched to the zip file vs API method?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or maybe just keep the load_json and load_df_to_pg tasks and get rid of the extract task?
"fda_enforcement", | ||
con=engine, | ||
schema="datasource", | ||
if_exists="append", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now that we are kind of loading the full file every time, we probably want to change this to "replace" - "append" was used originally b/c we were loading week by week.
Explanation
Made the following changes:
Did not change all DAGs, remaining DAGs that to be updated includes:
Tests
Tested each touched DAG with the following results: (:heavy_check_mark: means all tasks ran)