Add DB1 and DB2 report processing to SUSHI #31
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 7 commits
April 20, 2016 17:42
DB reports are different because they have multiple metrics on multiple lines, and SUSHI protocol provides limited guarantees on how they will be formatted. Catch month values in a dict with metrics as keys, then send each as a CounterDatabase object
Provide a mapping from SUSHI codes to COUNTER names for DB reports. If parsing a pre-formatted COUNTER report, it will fall back to whatever is already in the report.
SUSHI protocol does not require transmission of fields with zero use. Ensures months with no data are inserted with a 0, then sorted. Calls from CounterDatabase.as_generic to accurately write as a COUNTER compliant report.
SUSHI protocol does not guarantee fields with zero use are in a report. If a field has no use for a given period, it might not show up in the report at all. If a report is a datbase report, check each database to ensure it has all metrics and insert them with a 0 for the first month of the reporting period. After all metrics have been established, sort them so they write to file in the correct order (sort order will be maintained after the subsequent title sort)
TestConvertRawDatabase tests that all information is correct and in the correct order for a DB report. Utilizes sushi_simple_db1.xml which is in part based on formatting from ProQuest SUSHI reports TestRawDatabaseWithMissingData tests that, on output, SUSHI data is correct and full even if zero-value fields were not sent. January is missing data and there are no records for 'record_view' in the data source. Utilizes sushi_db1_missing_record_view.xml which is in part based on formatting from EBSCO SUSHI reports.
|
LGTM; I'm not too worried about the coveralls complaint. I'll play around with it a bit tomorrow morning and most likely merge it. Thank you! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added code to generically catch metrics from SUSHI responses and create CounterDatabase instances for each. Required two new methods, one to CounterDatabase parent class CounterEresource to check that all months in reporting period are present and add them if not. The other to CounterReport to check that all metrics required by COUNTER are present and add them if not. Unit tests and data files are provided for processing the reports, can provide for individual methods if requested.