DataCite Query Pagination Support #263
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 6 commits
October 4, 2021 16:13
… record index numbers with warnings when an optional field cannot be parsed
…initialize_production_deployment.py
nutjob4life
approved these changes
Oct 4, 2021
|
|
||
| # Append current results to full set returned | ||
| result = json.loads(datacite_response.text) | ||
| data.extend(result['data']) |
There was a problem hiding this comment.
Will there ever be a concern that data grows overly large if, say, the totalPages is also overly large? Should we be using a generator style pattern here to compensate instead of gathering and returning the entire kit & kaboodle?
There was a problem hiding this comment.
I would say this is not currently a concern based on two things:
- Based on the performance I observed importing all 1300+ records currently available on DataCite (which is only about a page and half):
INFO __main__:main DOI import complete in 23.26 seconds.
INFO __main__:main Num records found: 1390
INFO __main__:main Num records processed: 1390
INFO __main__:main Num records written: 1390
INFO __main__:main Num records skipped: 0
- This query function is only used right now for the local database importation script, so for the time being we shouldn't have to worry about an end-user of the service/UI running into some sort of performance bottleneck because we slurp all available pages.
|
Thanks @collinss-jpl! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🗒️ Summary
This PR adds support for DataCite's pagination scheme to the query function used to pull down existing DOI records from a DataCite server. This functionality should have been included with the original addition of DataCite support, but was overlooked.
In addition to pagination support, this branch adds several other small improvements, including:
⚙️ Test Data and/or Report
One of the following should be included here:
A unit test has been added for the pagination support added to the DataCite web client.
test.txt
♻️ Related Issues
Fixes #261