Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-23699: Update fgcmcal default config format #30

Merged
merged 3 commits into from
Mar 10, 2020
Merged

DM-23699: Update fgcmcal default config format #30

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ed69dff
Change list config fields to dicts.
erykoff Mar 3, 2020
e38fc9c
Add warnings when visits could not be calibrated.
erykoff Mar 6, 2020
11b7162
Add dict field validations and tests
erykoff Mar 10, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion python/lsst/fgcmcal/fgcmCalibrateTract.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,8 @@ def setDefaults(self):

self.fgcmBuildStars.checkAllCcds = False
self.fgcmBuildStars.densityCutMaxPerPixel = 10000
self.fgcmFitCycle.useRepeatabilityForExpGrayCuts = [True]
self.fgcmFitCycle.useRepeatabilityForExpGrayCutsDict = {b: True for
b in self.fgcmFitCycle.bands}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

b -> band would be clearer

self.fgcmFitCycle.quietMode = True
self.fgcmOutputProducts.doReferenceCalibration = False
self.fgcmOutputProducts.doRefcatOutput = False
Expand Down
200 changes: 161 additions & 39 deletions python/lsst/fgcmcal/fgcmFitCycle.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,13 +69,30 @@ class FgcmFitCycleConfig(pexConfig.Config):
"and matched band-by-band."),
dtype=int,
default=(0,),
optional=True,
deprecated=("This field is no longer used, and has been deprecated by DM-23699. "
"It will be removed after v20. Use fitBands instead."),
)
fitBands = pexConfig.ListField(
doc=("Bands to use in atmospheric fit. Other bands will have atmosphere constrained "
"from observations in the other bands on the same night."),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stutter on “other bands”...replace second with “those listed here”/“the bands used in the fit”?

State that this must be a subset of config.bands & validate?

Unrelated: what is the meaning of the default of ("NO_DATA",) for bands above?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All these fields are validated immediately by the fgcm code, and that's why I haven't put in redundant validations in fgcmcal. However, some basic validations would not be a bad idea. As for the NO_DATA, this was cargo-culted from somewhere...

dtype=str,
default=[],
)
requiredFlag = pexConfig.ListField(
doc=("Flag for which bands are required for a star to be considered a calibration "
"star in the FGCM fit. Typically this should be the same as fitFlag. Must "
"be same length as config.bands, and matched band-by-band."),
dtype=int,
default=(0,),
optional=True,
deprecated=("This field is no longer used, and has been deprecated by DM-23699. "
"It will be removed after v20. Use requiredBands instead."),
)
requiredBands = pexConfig.ListField(
doc="Bands that are required for a star to be considered a calibration star.",
dtype=str,
default=[],
)
filterMap = pexConfig.DictField(
doc="Mapping from 'filterName' to band.",
Expand Down Expand Up @@ -138,6 +155,15 @@ class FgcmFitCycleConfig(pexConfig.Config):
doc="Compute superstar flat on sub-ccd scale",
dtype=bool,
default=True,
optional=True,
deprecated=("This field is no longer used, and has been deprecated by DM-23699. "
"It will be removed after v20. Use superStarSubCcdDict instead."),
)
superStarSubCcdDict = pexConfig.DictField(
Copy link
Contributor

@laurenam laurenam Mar 10, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You now explicitly include (append) “Dict” in the name for DictFields, but don’t append “List” for ListFields (just food for thought from an OCD-leaning personality 😉 )

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sigh.

doc="Per-band specification on whether to compute superstar flat on sub-ccd scale.",
keytype=str,
itemtype=bool,
default={},
)
superStarSubCcdChebyshevOrder = pexConfig.Field(
doc=("Order of the 2D chebyshev polynomials for sub-ccd superstar fit. "
Expand All @@ -161,6 +187,15 @@ class FgcmFitCycleConfig(pexConfig.Config):
doc="Compute CCD gray terms on sub-ccd scale",
dtype=bool,
default=False,
optional=True,
deprecated=("This field is no longer used, and has been deprecated by DM-23699. "
"It will be removed after v20. Use ccdGraySubCcdDict instead."),
)
ccdGraySubCcdDict = pexConfig.DictField(
doc="Per-band specification on whether to compute ccd gray on sub-ccd scale.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“gray” what?

keytype=str,
itemtype=bool,
default={},
)
ccdGraySubCcdChebyshevOrder = pexConfig.Field(
doc="Order of the 2D chebyshev polynomials for sub-ccd gray fit.",
Expand Down Expand Up @@ -271,12 +306,32 @@ class FgcmFitCycleConfig(pexConfig.Config):
"Must be same length as config.bands, and matched band-by-band."),
dtype=float,
default=(0.0,),
optional=True,
deprecated=("This field is no longer used, and has been deprecated by DM-23699. "
"It will be removed after v20. Use expGrayPhotometricCutDict instead."),
)
expGrayPhotometricCutDict = pexConfig.DictField(
doc=("Per-band specification on maximum (negative) exposure gray for a visit to be "
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“gray” what?

"considered photometric. Must have one entry per band."),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this get checked in a validate on this config Task? And should you also check the validate in the tests (i.e. include some configs that don’t comply & assert that they fail)?

keytype=str,
itemtype=float,
default={},
)
expGrayHighCut = pexConfig.ListField(
doc=("Maximum (positive) exposure gray for a visit to be considered photometric. "
"Must be same length as config.bands, and matched band-by-band."),
dtype=float,
default=(0.0,),
optional=True,
deprecated=("This field is no longer used, and has been deprecated by DM-23699. "
"It will be removed after v20. Use expGrayHighCutDict instead."),
)
expGrayHighCutDict = pexConfig.DictField(
doc=("Per-band specification on maximum (positive) exposure gray for a visit to be "
"considered photometric. Must have one entry per band."),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto & ditto 😉

keytype=str,
itemtype=float,
default={},
)
expGrayRecoverCut = pexConfig.Field(
doc=("Maximum (negative) exposure gray to be able to recover bad ccds via interpolation. "
Expand All @@ -289,6 +344,16 @@ class FgcmFitCycleConfig(pexConfig.Config):
doc="Maximum exposure variance to be considered possibly photometric",
dtype=float,
default=0.0005,
optional=True,
deprecated=("This field is no longer used, and has been deprecated by DM-23699. "
"It will be removed after v20. Use expVarGrayPhotometricCutDict instead."),
)
expVarGrayPhotometricCutDict = pexConfig.DictField(
doc=("Per-band specification on maximum exposure variance to be considered possibly "
"photometric."),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any guidance for another user on how to select a reasonable value here (and for all the other empty defaulted threshold configs)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have added a bunch of defaults to setDefaults

keytype=str,
itemtype=float,
default={},
)
expGrayErrRecoverCut = pexConfig.Field(
doc=("Maximum exposure gray error to be able to recover bad ccds via interpolation. "
Expand All @@ -312,12 +377,26 @@ class FgcmFitCycleConfig(pexConfig.Config):
"If set, must be same length as config.bands, and matched band-by-band."),
dtype=float,
default=[],
optional=True,
deprecated=("This field is no longer used, and has been deprecated by DM-23699. "
"It will be removed after v20. Use aperCorrInputSlopeDict instead."),
)
aperCorrInputSlopeDict = pexConfig.DictField(
doc=("Per-band specification of aperture correction input slope parameters. These "
"are used on the first fit iteration, and aperture correction parameters will "
"be updated from the data if config.aperCorrFitNBins > 0. It is recommended "
"to set this when there is insufficient data to fit the parameters (e.g. "
"tract mode)."),
keytype=str,
itemtype=float,
default={},
)
sedFudgeFactors = pexConfig.ListField(
doc=("Fudge factors for computing linear SED from colors. Must be same length as "
"config.bands, and matched band-by-band."),
dtype=float,
default=(0,),
optional=True,
deprecated=("This field has been deprecated and will be removed after v20. "
"Please use sedSlopeTermMap and sedSlopeMap."),
)
Expand All @@ -339,6 +418,16 @@ class FgcmFitCycleConfig(pexConfig.Config):
"May be 1 element (same for all bands) or the same length as config.bands."),
dtype=float,
default=(0.05,),
optional=True,
deprecated=("This field is no longer used, and has been deprecated by DM-23699. "
"It will be removed after v20. Use sigFgcmMaxEGrayDict instead."),
)
sigFgcmMaxEGrayDict = pexConfig.DictField(
doc=("Per-band specification for maximum (absolute) gray value for observations in "
"sigma_fgcm."),
keytype=str,
itemtype=float,
default={},
)
ccdGrayMaxStarErr = pexConfig.Field(
doc="Maximum error on a star observation to use in ccd gray computation",
Expand All @@ -350,6 +439,16 @@ class FgcmFitCycleConfig(pexConfig.Config):
"May be 1 element (same for all bands) or the same length as config.bands."),
dtype=float,
default=(1.0, ),
optional=True,
deprecated=("This field is no longer used, and has been deprecated by DM-23699. "
"It will be removed after v20. Use approxThroughputDict instead."),
)
approxThroughputDict = pexConfig.DictField(
doc=("Per-band specification of the approximate overall throughput at the start of "
"calibration observations."),
keytype=str,
itemtype=float,
default={},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it required that this be set & have any entry for each band?

)
sigmaCalRange = pexConfig.ListField(
doc="Allowed range for systematic error floor estimation",
Expand Down Expand Up @@ -395,6 +494,14 @@ class FgcmFitCycleConfig(pexConfig.Config):
doc="Band indices to use to split stars by color",
dtype=int,
default=None,
optional=True,
deprecated=("This field is no longer used, and has been deprecated by DM-23699. "
"It will be removed after v20. Use colorSplitBands instead."),
)
colorSplitBands = pexConfig.ListField(
doc="Band names to use to split stars by color. Must have 2 entries.",
dtype=str,
default=None,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 and only 2? validate?

)
modelMagErrors = pexConfig.Field(
doc="Should FGCM model the magnitude errors from sky/fwhm? (False means trust inputs)",
Expand Down Expand Up @@ -445,6 +552,16 @@ class FgcmFitCycleConfig(pexConfig.Config):
"May be 1 element (same for all bands) or the same length as config.bands."),
dtype=bool,
default=(False,),
optional=True,
deprecated=("This field is no longer used, and has been deprecated by DM-23699. "
"It will be removed after v20. Use useRepeatabilityForExpGrayCutsDict instead."),
)
useRepeatabilityForExpGrayCutsDict = pexConfig.DictField(
doc=("Per-band specification on whether to use star repeatability (instead of exposures) "
"for computing photometric cuts. Recommended for tract mode or bands with few visits."),
keytype=str,
itemtype=bool,
default={},
)
autoPhotometricCutNSig = pexConfig.Field(
doc=("Number of sigma for automatic computation of (low) photometric cut. "
Expand Down Expand Up @@ -785,12 +902,17 @@ def _fgcmFitCycle(self, butler):
# Output the config for the next cycle
# We need to make a copy since the input one has been frozen

updatedPhotometricCutDict = {b: float(fgcmFitCycle.updatedPhotometricCut[i]) for
i, b in enumerate(self.config.bands)}
updatedHighCutDict = {b: float(fgcmFitCycle.updatedHighCut[i]) for
i, b in enumerate(self.config.bands)}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

b -> band?

outConfig = copy.copy(self.config)
outConfig.update(cycleNumber=(self.config.cycleNumber + 1),
precomputeSuperStarInitialCycle=False,
freezeStdAtmosphere=False,
expGrayPhotometricCut=fgcmFitCycle.updatedPhotometricCut,
expGrayHighCut=fgcmFitCycle.updatedHighCut)
expGrayPhotometricCutDict=updatedPhotometricCutDict,
expGrayHighCutDict=updatedHighCutDict)
configFileName = '%s_cycle%02d_config.py' % (outConfig.outfileBase,
outConfig.cycleNumber)
outConfig.save(configFileName)
Expand Down Expand Up @@ -878,9 +1000,9 @@ def _loadParameters(self, butler):

inParInfo = np.zeros(1, dtype=[('NCCD', 'i4'),
('LUTFILTERNAMES', parLutFilterNames.dtype.str,
parLutFilterNames.size),
('FITBANDS', parFitBands.dtype.str, parFitBands.size),
('NOTFITBANDS', parNotFitBands.dtype.str, parNotFitBands.size),
(parLutFilterNames.size, )),
('FITBANDS', parFitBands.dtype.str, (parFitBands.size, )),
('NOTFITBANDS', parNotFitBands.dtype.str, (parNotFitBands.size, )),
('LNTAUUNIT', 'f8'),
('LNTAUSLOPEUNIT', 'f8'),
('ALPHAUNIT', 'f8'),
Expand All @@ -900,74 +1022,74 @@ def _loadParameters(self, butler):
inParInfo['HASEXTERNALPWV'] = parCat['hasExternalPwv']
inParInfo['HASEXTERNALTAU'] = parCat['hasExternalTau']

inParams = np.zeros(1, dtype=[('PARALPHA', 'f8', parCat['parAlpha'].size),
('PARO3', 'f8', parCat['parO3'].size),
inParams = np.zeros(1, dtype=[('PARALPHA', 'f8', (parCat['parAlpha'].size, )),
('PARO3', 'f8', (parCat['parO3'].size, )),
('PARLNTAUINTERCEPT', 'f8',
parCat['parLnTauIntercept'].size),
(parCat['parLnTauIntercept'].size, )),
('PARLNTAUSLOPE', 'f8',
parCat['parLnTauSlope'].size),
(parCat['parLnTauSlope'].size, )),
('PARLNPWVINTERCEPT', 'f8',
parCat['parLnPwvIntercept'].size),
(parCat['parLnPwvIntercept'].size, )),
('PARLNPWVSLOPE', 'f8',
parCat['parLnPwvSlope'].size),
(parCat['parLnPwvSlope'].size, )),
('PARLNPWVQUADRATIC', 'f8',
parCat['parLnPwvQuadratic'].size),
(parCat['parLnPwvQuadratic'].size, )),
('PARQESYSINTERCEPT', 'f8',
parCat['parQeSysIntercept'].size),
(parCat['parQeSysIntercept'].size, )),
('COMPQESYSSLOPE', 'f8',
parCat['compQeSysSlope'].size),
(parCat['compQeSysSlope'].size, )),
('PARFILTEROFFSET', 'f8',
parCat['parFilterOffset'].size),
(parCat['parFilterOffset'].size, )),
('PARFILTEROFFSETFITFLAG', 'i2',
parCat['parFilterOffsetFitFlag'].size),
(parCat['parFilterOffsetFitFlag'].size, )),
('PARRETRIEVEDLNPWVSCALE', 'f8'),
('PARRETRIEVEDLNPWVOFFSET', 'f8'),
('PARRETRIEVEDLNPWVNIGHTLYOFFSET', 'f8',
parCat['parRetrievedLnPwvNightlyOffset'].size),
(parCat['parRetrievedLnPwvNightlyOffset'].size, )),
('COMPABSTHROUGHPUT', 'f8',
parCat['compAbsThroughput'].size),
(parCat['compAbsThroughput'].size, )),
('COMPREFOFFSET', 'f8',
parCat['compRefOffset'].size),
(parCat['compRefOffset'].size, )),
('COMPREFSIGMA', 'f8',
parCat['compRefSigma'].size),
(parCat['compRefSigma'].size, )),
('COMPMIRRORCHROMATICITY', 'f8',
parCat['compMirrorChromaticity'].size),
(parCat['compMirrorChromaticity'].size, )),
('MIRRORCHROMATICITYPIVOT', 'f8',
parCat['mirrorChromaticityPivot'].size),
(parCat['mirrorChromaticityPivot'].size, )),
('COMPAPERCORRPIVOT', 'f8',
parCat['compAperCorrPivot'].size),
(parCat['compAperCorrPivot'].size, )),
('COMPAPERCORRSLOPE', 'f8',
parCat['compAperCorrSlope'].size),
(parCat['compAperCorrSlope'].size, )),
('COMPAPERCORRSLOPEERR', 'f8',
parCat['compAperCorrSlopeErr'].size),
(parCat['compAperCorrSlopeErr'].size, )),
('COMPAPERCORRRANGE', 'f8',
parCat['compAperCorrRange'].size),
(parCat['compAperCorrRange'].size, )),
('COMPMODELERREXPTIMEPIVOT', 'f8',
parCat['compModelErrExptimePivot'].size),
(parCat['compModelErrExptimePivot'].size, )),
('COMPMODELERRFWHMPIVOT', 'f8',
parCat['compModelErrFwhmPivot'].size),
(parCat['compModelErrFwhmPivot'].size, )),
('COMPMODELERRSKYPIVOT', 'f8',
parCat['compModelErrSkyPivot'].size),
(parCat['compModelErrSkyPivot'].size, )),
('COMPMODELERRPARS', 'f8',
parCat['compModelErrPars'].size),
(parCat['compModelErrPars'].size, )),
('COMPEXPGRAY', 'f8',
parCat['compExpGray'].size),
(parCat['compExpGray'].size, )),
('COMPVARGRAY', 'f8',
parCat['compVarGray'].size),
(parCat['compVarGray'].size, )),
('COMPNGOODSTARPEREXP', 'i4',
parCat['compNGoodStarPerExp'].size),
(parCat['compNGoodStarPerExp'].size, )),
('COMPSIGFGCM', 'f8',
parCat['compSigFgcm'].size),
(parCat['compSigFgcm'].size, )),
('COMPSIGMACAL', 'f8',
parCat['compSigmaCal'].size),
(parCat['compSigmaCal'].size, )),
('COMPRETRIEVEDLNPWV', 'f8',
parCat['compRetrievedLnPwv'].size),
(parCat['compRetrievedLnPwv'].size, )),
('COMPRETRIEVEDLNPWVRAW', 'f8',
parCat['compRetrievedLnPwvRaw'].size),
(parCat['compRetrievedLnPwvRaw'].size, )),
('COMPRETRIEVEDLNPWVFLAG', 'i2',
parCat['compRetrievedLnPwvFlag'].size),
(parCat['compRetrievedLnPwvFlag'].size, )),
('COMPRETRIEVEDTAUNIGHT', 'f8',
parCat['compRetrievedTauNight'].size)])
(parCat['compRetrievedTauNight'].size, ))])

inParams['PARALPHA'][:] = parCat['parAlpha'][0, :]
inParams['PARO3'][:] = parCat['parO3'][0, :]
Expand Down