Skip to content

Conversation

@jlchang
Copy link
Contributor

@jlchang jlchang commented Sep 23, 2020

Users of the metadata convention reported failed validation Notifier emails that had stack traces with "unexpected exit status 65 was not ignored" and no information on how to fix their metadata files.

This simple PR changes the metadata convention from using the dev_logger to using the user_logger.

The investigation for this ticket uncovered two issues:

  1. ingest_pipeline is treating numeric array-based values as invalid non-numeric data and rejecting the file as format invalid. Created SCP-2756 to deal address this problem. Tests for array-based validation did not reveal this problem because the tests were not using the newly abstracted file format checking functions.
  2. running ingest_pipeline.py produces results different from validate_metadata.py because format validation has been abstracted to a higher level and ingest_pipeline catches format errors that occur upstream of validate_metadata activity. Fixing this issue will be much more straightforward when the issue above has been resolved and the metadata validation tests can use the abstracted file format checks.

remove unnecessary output files
once numeric array-based metadata ingest issue resolved
@bistline bistline self-requested a review September 23, 2020 18:58
Copy link
Contributor

@bistline bistline left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, just a question about additional logging.


if issue_type == "error":
errfile.write(msg + "\n")
dev_logger.error(msg)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason why we wouldn't want to log this in the dev logger?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sending an error to the user_logger will automatically also send it to the dev logger so we're covered.

@jlchang jlchang merged commit 779c4f6 into development Sep 24, 2020
@jlchang jlchang deleted the jlc_swallowed_errors branch October 8, 2020 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants