DM-5120: Add intelligence to validate_drp so it does "A Reasonable Thing" on an unknown output repo
#10
Conversation
Return only dataIds with 'src' and 'calexp' on disk. use queryMetadata() to retrieve the filter information in case the dataId from Butler.Subset doesn't include it
Update README.md and examples for new calling format for `validateDrp.py`
| calib = afwImage.Calib(butler.get("calexp_md", vId, immediate=True)) | ||
| except TypeError as te: | ||
| print(te) | ||
| continue |
There was a problem hiding this comment.
Why would this throw a TypeError? (I'm not claiming it won't or can't, I'm just a little surprised.) If it happens, can we do something more informative than just printing the exception? (Even adding "Skipping to next data ID" might be useful, for e.g.).
Also, this code is added in 8fc82e0, which is fundamentally about a different issue (adding the no-throw decorator, below). Can we split it into its own commit with a message that explains why it's necessary?
There was a problem hiding this comment.
Good question. Now documented in code comment below.
# DECam images that haven't been properly reformatted
# can trigger a TypeError because of a residual FITS header
# LTV2 which is a float instead of the expected integer.
# This generates an error of the form:
#
# lsst::pex::exceptions::TypeError: 'LTV2 has mismatched type'
#
# See, e.g., DM-2957 for details.
Also added catching FitsError exception when reading FITS files.
There was a problem hiding this comment.
Didn't DM-4133 solve the LTV1/2 mismatched type issue? Or is this from something else?
There was a problem hiding this comment.
Yes, this is from the same problem solved by DM-4133. Unfortunately problems in reduced data persist even after code has been updated. I think we'll unfortunately be seeing such issues and accumulating a bit of cruft like this as we have to deal with more and more actual data.
When I ran validateDrp.py on the DECam COSMOS data as processed as NCSA, there were still a number of files with this issue. This catch is designed to skip those and proceed onward.
MWV
On Mar 3, 2016, at 10:29, hchiang2 notifications@github.com wrote:
In python/lsst/validate/drp/validate.py:
@@ -88,17 +92,23 @@ def loadAndMatchData(repo, visitDataIds,
srcVis = SourceCatalog(newSchema)for vId in visitDataIds:
calib = afwImage.Calib(butler.get("calexp_md", vId, immediate=True))calib.setThrowOnNegativeFlux(False)try:calib = afwImage.Calib(butler.get("calexp_md", vId, immediate=True))except TypeError as te:print(te)continueDidn't DM-4133 solve the LTV1/2 mismatched type issue? Or is this from something else?
—
Reply to this email directly or view it on GitHub.
If the configFile has only validation parameters specified (e.g., good_mag_limit) then use util.discoverDataIds to identify the dataIds to analyze. Update naming of visitDataIds->dataIds.
Use afwImageUtils.CalibNoThrow context manager instead of explicitly setting the state in the Calib object Add catching FitsError + explain TypeError reading calexp Clarify error messages from missing calibration. Explain why the TypeError needs to be caught when reading LTV2 keywords in DECam images that haven't been fully sanitized.
The caveat is that this is single-node, load-everything-in-memory, and then match. This is infeasible at scale. Specifically, this currently fails on lsst-dev at more than ~500 catalogs in a band.
validateDrp.py Cfht/output
will now do "something reasonable" with no further configuration information needed.