DM-32615: Move jointcal cfht_minimal tests to gen3 #207
Conversation
Copied from testdata_jointcal/cfht, so that all the relevant fields are updated for the postprocess script.
parejkoj
force-pushed
the
tickets/DM-32615
branch
from
7638fb5
to
920b3b3
Compare
| from lsst.daf.butler.script import ingest_files | ||
| import lsst.obs.base | ||
|
|
||
| repopath = "cfht_minimal/repo" |
There was a problem hiding this comment.
os.path.join(os.environ["JOINTCAL_DIR"], 'tests/data/cfht_minimal/repo') doesn't work? (If not here, then in the butler.import_ call?) But also, I don't see absolute paths in sdss-dr9-fink-v5b.ecsv.
There was a problem hiding this comment.
I couldn't figure it out: the paths in the ecsv file are all relative, so they should have been relative to whatever prefix= I passed in to ingest_files. I believe that I want the repopath to be relative too, so that everything inside the butler is relative to that (though I can't confirm that changing it the way you suggest would cause butler-internal paths to be absolute).
| with self.assertRaisesRegex(RuntimeError, "No data to process"): | ||
| jointcal.JointcalTask.parseAndRun(args=args, doReturnResults=True, config=self.config) | ||
|
|
||
| def test_jointcalTask_fails_no_raise(self): |
There was a problem hiding this comment.
Is it safe to remove this test completely?
There was a problem hiding this comment.
Yes: this is testing gen2-only functionality here. The test_jointcalTask_fails_raise test demonstrates that JointcalTask will raise if there is no data, but whether it raises in a pipetask context, or just goes on to the next quantum and logs a failure is entirely up to the pipetask configuration and not something I can test within jointcal itself.
| instrInstance.register(butler.registry) | ||
|
|
||
| graph = butler.registry.dimensions.extract(["htm7"]) | ||
| datasetType = lsst.daf.butler.DatasetType("sdss_dr9_fink_v5b", graph, |
There was a problem hiding this comment.
Seeing how often sdss_dr9_fink_v5b appears everywhere, would it make sense to define a refcat_default = "sdss_dr9_fink_v5b" somewhere in the package?
There was a problem hiding this comment.
In the package, or just in this file? The sdss refcats are really only used in a couple of tests, so they definitely shouldn't be package-wide defaults. Those are Gaia DR2 for astrometry and PS1 for photometry.
There was a problem hiding this comment.
Oh, fair enough. Maybe we should have some refcat names defined in a package somewhere because it seems like it'll be a huge pain if we ever want to update them en masse.
There was a problem hiding this comment.
Eh, these particular refcats will probably never get updated, since they are each for a specific testing purpose. If we add e.g. a catalog with proper motion covariances or something, I'll just add a specific test for that catalog.
For non-jointcal packages, I think we'll be consolidating some of those names as part of the drp_pipe work, and I plan to tackle DM-27013 over the break, which should help.
Gen3 jointcal needs the parquet tables, and we need to tweak how they are made because this is much older data that is missing many expected fields.
This file is made from the testdata_jointcal/cfht/exports.yaml, modified for the layout of the gen2 files in cfht_minimal.
parejkoj
force-pushed
the
tickets/DM-32615
branch
from
74f5501
to
1ac8998
Compare
These files were generated with the postprocess_cfht_minimal.py script.
Remove no_raise tests, as that is all handled internal to pipetask now; keep the test of raising when there is no data since that is the correct behavior now, always. Add minSnr config for cfht_minimal tests. Modify cfht_minimal-config to reflect the fact that the catalogs come from much older processing.
Add sdss refcat .ecsv ingest file Update test data readme
tests/data/cfht_minimal now has a working stripped down gen3 repo to use for these tests. Cleanup some "ccd"->"detector" names Remove unnecessary gen2 butler import from test_photometryModel.
parejkoj
force-pushed
the
tickets/DM-32615
branch
from
1ac8998
to
7e71501
Compare
No description provided.