New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update NYPL DAG to use API v2 #3774
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome that the update only took ~22 lines of code! Tested fine locally.
Based on the high urgency of this PR, the following reviewers are being gently reminded to review this PR: @krysal Excluding weekend1 days, this PR was ready for review 2 day(s) ago. PRs labelled with high urgency are expected to be reviewed within 2 weekday(s)2. @AetherUnbound, if this PR is not ready for a review, please draft it to prevent reviewers from getting further unnecessary pings. Footnotes
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's nice that it was a simple update!
"type_of_resource": "cartographic", | ||
"physical_description": "4 polyester film encapsulations, some containing 2 sheets back-to-back. " | ||
"Accompanying text formatted as 1 large sheet (46 x 59 cm), in one of " | ||
"the encapsulations.", | ||
}, | ||
"raw_tags": ["Census districts"], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great to send this field to the correct place now!
Fixes
Fixes #2747 by @krysal
Description
This PR updates the NYPL DAG to use the new v2 API. Fortunately almost nothing changed with the actual API interaction details! So this became a very straightforward change. The only major change was putting the tags in the right place!
I also added the option to pass arguments into the catalog's
ipython
recipe.Testing Instructions
AIRFLOW_VAR_API_KEY_NYPL
in the.env
AIRFLOW_VAR_INGESTION_LIMIT
(note that this DAG is slow and will take about 3 hours to run end-to-end for ~3k results).Checklist
Update index.md
).main
) or a parent feature branch.Developer Certificate of Origin
Developer Certificate of Origin