DM-29348: Add ability to run global fgcmcal in a single pipeline. #59
Conversation
erykoff
force-pushed
the
tickets/DM-29348
branch
3 times, most recently
from
50094d7
to
dd0c327
Compare
| # Run multiple cycles at once. | ||
| config = copy.copy(self.config) | ||
| config.update(cycleNumber=0) | ||
| for cycle in range(self.config.multipleCyclesFinalCycleNumber + 1): |
There was a problem hiding this comment.
So we will be doing multipleCyclesFinalCycleNumber+1 cycles, right? Since initial cycle is number 0?
I'm guessing the reason we want the min field in the config to be 2 is that we have at least one non-initial, non-final cycle?
There was a problem hiding this comment.
Yes, the initial cycle is 0, that is done with limited parameters free. The final cycle (as you say, n+1) is a clean-up with no parameters free. So that makes the min field 2. I'll add comments/docs to this effect.
| @@ -85,31 +85,31 @@ class FgcmOutputProductsConnections(pipeBase.PipelineTaskConnections, | |||
| deferLoad=True, | |||
| ) | |||
|
|
|||
| fgcmVisitCatalog = connectionTypes.PrerequisiteInput( | |||
| fgcmVisitCatalog = connectionTypes.Input( | |||
There was a problem hiding this comment.
Why are we changing all these to just Inputs? As far as I can tell, some are still always used (unlike eg fgcmAtmosphereParameters that may not be needed depending on the config) here and in fgcmFitCycle...?
There was a problem hiding this comment.
Previously, the three tasks (build stars; fit; output products) all had to be done separately. And I believe a PrerequisiteInput is faster for the the quantum graph generator to find (or not find). But now I have the ability to link them all into one pipeline, so the fgcmVisitCatalog doesn't actually exist prior to the start of the pipeline run, so it must be a regular Input that is constrained by the quantum graph generator.
| fgcmBuildStarsConfig = butlerQC.get(inputRefs.fgcmBuildStarsTableConfig) | ||
| physicalFilterMap = fgcmBuildStarsConfig.physicalFilterMap | ||
|
|
||
| if self.config.doComposeWcsJacobian and not fgcmBuildStarsConfig.doApplyWcsJacobian: | ||
| raise RuntimeError("Cannot compose the WCS jacobian if it hasn't been applied " | ||
| "in fgcmBuildStarsTask.") | ||
| if not self.config.doComposeWcsJacobian and fgcmBuildStarsConfig.doApplyWcsJacobian: | ||
| self.log.warn("Jacobian was applied in build-stars but doComposeWcsJacobian is not set.") |
There was a problem hiding this comment.
We seem to be unable to read in the configs as regular inputs, and these checks seemed extraneous. But my intention is to add these to the gen3 pipeline "contracts". The pipeline itself isn't anywhere yet since it doesn't quite fit into the current framework; I'll ticket that separately.
| @@ -1,12 +1,6 @@ | |||
| import lsst.fgcmcal as fgcmcal | |||
|
|
|||
| config.outfileBase = 'TestFgcm' | |||
| # The unused z-band is here to test the case that extra filters | |||
There was a problem hiding this comment.
We're still getting that extra-filter test with HSC_FILTER_DEFINITIONS.physical_to_band, right?
FGCM can now be configured to run the full pipeline (build stars, fit multiple cycles, and output) in one run. The gen3 testing has also been refactored to use the same repository for all the tests.
erykoff
force-pushed
the
tickets/DM-29348
branch
from
dd0c327
to
cff9057
Compare
FgcmFitCycleTaskhas been updated to allow multiple fit cycles to be run at once. This, in turn, allows the full fgcmcal (build stars; multiple fit cycles; output photoCalibs) to be run as a single pipeline. The gen3 tests now share a common test repo.