DM-21221: PTC task should produce a linearity model

[plazas]

No description provided.

[czwa]

Does it makes sense to make lookup tables that are this large?

[plazas]

I also wondered about this. I wanted to cover, at least a priori, the possible ADU range for LSSTCam (that goes to 18 bits as opposed to other cameras, such as DECam, that goes to 16 bits). In practice, I would think there will be some cutoff (< 2**18) in the maximum ADU (for example, the file "obs_lsst/policy/lsstCam.yaml" has "linearityMax : 142857" ). Should perhaps this value ("numberAduValues") be set as a config parameter instead?

[czwa]

I think it needs to be configurable at least.

[czwa]

I have concerns:

• Fitting two polynomials seems like a waste. I get that we want a squared term for a LinearizeSquared, but if the linearization doesn't match that functional form, that's important to know. I would think that fitting a higher order polynomial (cubic or whatever), and then confirming that all terms k[3:] are ~zero would be better. This would ensure that both of the forms returned yield the same science results as well (if the linearization is fit well by LinearizeSquared, then the image difference in using that rather than a lookup table form that contains a full n-th order polynomial fit with small higher order terms will also be small).
• I understand that we don't yet have a generic polynomial linearizer, but I think being ready to support that would be better than converting the polynomial into a lookup table.
• Can we also have a way to put an empirical non-parametric model into the lookup table? I thought that was what that was for originally (I think only obs_decam uses that code, and if I remember the curves correctly, that has linearity models with discontinuous linear segments).

[plazas]

1. I agree and was debating about this. In the end, I decided to fit the quadratic polynomial too because I thought that "LinearizeSquared" assumed that this is exactly the case (i.e, the c0 coefficient comes from exactly a quadratic fit, not a polynomial of another order). However, as you point out, I can just stick to the general n-order polynomial, still approximate c0 to -k2/(k1^2), and check that the k[3:] are ~zero (issuing a warning if for some reason they are not).
2. Something for another ticket?
3. As in 2), perhaps this could be done in another ticket. I think you are correct, in the sense that, for example, we would like to use information from a calibration diode to correct for signal-chain non-linearity. For this ticket, we agreed (in Jira) that, for the moment, we would do something simple like taking the linear part of an n-degree polynomial as the linearity correction, and that we would put that information in a lookup table.

[czwa]

Ok, I think I wasn't sure how much had been delegated to the future/alternate linearity functions/etc.

[czwa]

Since this is a "human curated" parameter, would it make sense to persist the linearity as something human readable? I'm not sure what use cases you have for the PTC data, so I understand if these get read back in a lot that a pickle would be faster than yaml, but if this is just a set of values that are calculated and written and then used to construct the camera configuration, I would think yaml would be easier to use.

[plazas]

I still don't know how the c0 will be read in case that someone wants to use LinearizeSquared, and hence I don't know where it should be exactly stored and in which format We discussed this about in one of our meetings, I think, and I believe we agreed that, for the moment, we would just save it somewhere. Since the dataset structure was already in place, I decided to append the coefficient to it. I have used the PTC data, such as the gains, and the noise, to do comparisons with other codes (e.g., the BOT analyses), and pickles have worked fine so far.

[czwa]

Since writing this, I've discovered that yaml files aren't fully supported by butler yet (particularly in gen3). I am going to try to get that working soon, because I like having human readable things easily readable by humans. Pickle is fine for now.

[timj]

Can you let me know what you mean by "not fully supported". I use yaml in gen3 all the time.

[czwa]

I've had to add storageClass/formatter definitions to get yaml dict output to be understood (and I thought I'd rebased to ensure I had everything up to date).

[timj]

Ah, well, we may want to move this definition out of the daf_butler tests then:

storageClasses:
  StructuredDataDictYaml:
    pytype: dict

and add

StructuredDataDictYaml: lsst.daf.butler.formatters.yamlFormatter.YamlFormatter

to the config/datastores/formatter.yaml. We hadn't enabled simply python types by default and expected there to be more expressive types read from YAML files. The default YamlFormatter isn't super great at casting the dict returned to a new type (it does try and could be improved).

[czwa]

Pending remaining "Nl" name changes to be less confusing (even full cap NL would work).