Write metadata error messages to workspace bucket [SCP-1969] #47
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
eweitz
approved these changes
Nov 7, 2019
Member
eweitz
left a comment
eweitz
left a comment
There was a problem hiding this comment.
Looks good! Thanks for the walk-through.
bistline
approved these changes
Nov 12, 2019
knapii-developments
approved these changes
Nov 14, 2019
…-pipeline into jlc_error2bucket
Codecov Report
@@ Coverage Diff @@
## master #47 +/- ##
=========================================
+ Coverage 63.74% 63.8% +0.06%
=========================================
Files 11 11
Lines 1208 1235 +27
=========================================
+ Hits 770 788 +18
- Misses 438 447 +9
Continue to review full report at Codecov.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements writing metadata validation errors to a local file that is then delocalized to the google bucket belonging to the corresponding cell_metadata file.
Implementation notes:
• validation error output file (local or bucket) is overwritten with each new validation run
• added self.local_file_path (ingest_files.py, resolve_path) to address issue with opening metadata file located in google bucket when open_as "dataframe"
• delocalize_error_file and the conditions of its invocation in ingest_pipeline.py currently hard-coded specific to metadata ingest case. Refactoring will be needed if we're capturing ingest errors of other file types.
This PR also includes:
• updated metadata validation tests to conform to latest metadata convention (AMC_v1.1.3)
This fulfills SCP-1969