Batch processing logic for GLOS files #6
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds a method to process multiple subdirectories to be used with QARTOD, pushing files which have a relevant configuration onto a RQ jobs queue.
Allows empty string config values to not attempt a unit conversion in the NetCDF file. This was necessary for certain non-udunits conforming units present in NetCDF files such as "RFU" and "su". Also handles exceptions when a unit conversion is attempted but not successful.
Handles zero sized arrays which currently cause issues with some QARTOD tests. Also add times to parameters passed for spike test, as the method signature for the corresponding QARTOD test has changed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checks if GLOS files have already been QCed, going by existence of a variable. This is appropriate for batch processing files which have not already been QCed or are not in need of real-time QC as it will not attempt to QC when all given QC variables exist.