New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DM-28636: various improvements to help with shared repo setup #372
Conversation
Need to register the CALIBRATION collection before trying to certify into it.
This makes it a bit more possible for us to catch problems before we start writing to the Gen3 repo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks okay although I think I'd expect log messages for each dry run else clause to give some feedback as to what was being skipped.
@@ -119,16 +120,19 @@ def runRawIngest(self, pool=None): | |||
dataRefs = self.butler2.subset(self.task.config.rawDatasetType) | |||
dataPaths = getDataPaths(dataRefs) | |||
self.task.log.info("Ingesting raws from root %s into run %s.", self.root, self.task.raws.butler.run) | |||
self._rawRefs.extend(self.task.raws.run(dataPaths, pool=pool)) | |||
if not self.task.dry_run: | |||
self._rawRefs.extend(self.task.raws.run(dataPaths, pool=pool)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I'd expect dry run to issue a log message saying something like "Skipping raw ingest for {dataPaths}".
return | ||
dimensions = DimensionGraph(self.task.universe, names=["exposure"]) | ||
exposureDataIds = set(ref.dataId.subset(dimensions) for ref in self._rawRefs) | ||
self.task.log.info("Defining visits from exposures.") | ||
self.task.defineVisits.run(exposureDataIds, pool=pool) | ||
if not self.task.dry_run: | ||
self.task.defineVisits.run(exposureDataIds, pool=pool) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As above may a log message saying that N exposureDataIds would have had visits defined?
73afd3f
to
7a54d1e
Compare
Many other improvements have already been spun off into their own tickets. What remains here are miscellaneous small things that didn't seem to meet that threshold.