Determine list of image metadata keywords that should be in the primary HDU

[jchiang87]

This is to support jointcal processing. See LSSTDESC/DC2-production#275

[cwwalter]

Can we calculate this ourselves from pointing, or must we use the OpSim database? It's nice to not have to depend on having a database for standalone imSim usage.

[jchiang87]

We have a function to calculate it using the altitude info from the instance catalog (which also has an airmass entry that we don't use).

[cwwalter]

I've made this issue more general. In addition to Airmass which came up before (for jointcal and also for DCR correction), @rmjarvis has pointed out that gain is also important.

I will first try to see if there is a canonical list of what the real data is supposed to include. Then we can make a list of what should be added (either in imSim or by DM) and if we have any feedback of things that should be in the real data that are missing.

[jmeyers314]

I'd add
• parallactic angle
• temperature
• pressure
• water-vapor partial pressure
to the list if they're available, as these all also affect DCR.

[cwwalter]

Thank you Josh.

Others interested in metadata that they think is necessary please feel free to add to this thread. I did some research and talked to some of the DM group working on this issue (especially @ktlim). Here is a bit of what I learned:

• KT and others are working on making such a list of headers that should be included taking inputs from many stake holders. I have asked for us to also give input. It will eventually exist in LSE-400. So this is a good place to gather thoughts you would like people working on this to see.

My current rough understanding of the architecture is as follows and architecture experts may correct or expand this:

• Keywords can be added to files (or to databases that can later be accessed) in more than one way and at different times. The camera control system (CSS) can write a FITS file with internal camera information and also provide data to the header service (along with other slow-control systems like those for the telescope and environmental conditions). The header service will provide a header object, which will be passed to the archiver service which which can then add them to the files.

• Additionally other pieces of metadata may be added or corrected after the fact. The resulting data structure (sometimes referred to as "extended raw") will be a mixture of original headers and updated metadata as stored in a database and retrieved via the butler. This is described in more detail here: https://confluence.lsstcorp.org/display/DM/Image+Metadata+Flow#ImageMetadataFlow-TelemetryandEventData . For example:

In the process of generating Calibration Products or preparing for a
Data Release Production, new values for metadata items may be
generated from time to time as our understanding of them improves.
For example, we may update the conversion from voltage to physical
units for a telemetry item, change the method of interpolation for a
slowly-changing item, or change the method of averaging for a
rapidly-changing item.  Some simply incorrect values caused by
hardware or software failure may be replaced.  All changes of this
type will result in new versions of the image metadata in the raw
image metadata tables.  The Data Release Production, Calibration
Products Productions, and the image access services may all request
various versions of the image metadata: the version recorded at pixel
capture time ("original"), the latest and "best" version, or a
specific version from a particular (processing, not observation)
date/time.  This will be implemented in the Butler as a composite
dataset with pixels from a file and metadata from the database.

Other information can be added by DM as processing progresses. There will hopefully be a very preliminary list (subject to change at any time) on the time scale of a month.

I believe our goal is to write files that will include the information added by the entire system before any new information is added by DM. In other words: we should endeavor to make the amp files look as much like what comes out of the telescope system as a whole as possible. If we determine there are things we think should be included but is not currently envisioned we should suggest them.

[cwwalter]

Added from #166:

Metadata to add to the eimage and raw file headers:

DATE-OBS in eimage files.
Random seed value used for cosmic ray draws, read noise, and dark current.

[cwwalter]

We should also add imSim metadata to the headers. These could be configuration values but at a minimum we should be recording the imSim version string in the header so we can tell after the fact what version of the code made the file.

We might also consider recording associated software versions if we can access them.

[timj]

Please ensure you have INSTRUME and TELESCOP. DATE-OBS as you say, preferably DATE-END (could be the MJD- versions of those). OBSGEO-[XYZ] are part of the FITS standard and would be nice. It would be great if we could have a "SIMULATE = T" header (I've been talking to @ktlim and people in telescope and site about having a way to indicate that part of the system was in simulation). ORIGIN = 'IMSIM' would also be good.

[jchiang87]

@timj Is this everything? We've asked DM for help on desired header contents before. It would be good to head off further complaints by being able to implement the complete list.

[timj]

@jchiang87 I've been working with @menanteau on the AuxTel LATISS header (you can see the current state at https://github.com/lsst-dm/HeaderService/blob/master/etc/ATSCam/primary_hdu.header ) but we have nothing formally defined in LSE-400 yet. @ktlim is tasked with bringing that information together.

[cwwalter]

I believe we have now added what we can currently based on #192 and lsst-sims/legacy_sims_GalSimInterface#75 (which added the keywords requested by @timj) So. This issue will stay open since we have more to do but I am now removing the DC2 keyword.

[boutigny]

Following today's #desc-dm-dc2 meeting regarding the extra metadata that should be added in or derived from the raw header in order to run jointcal, I digged out the following from an old thread:

    _boresightRaDec = visitInfo->getBoresightRaDec();
    _airMass = visitInfo->getBoresightAirmass();
    _mjd = visitInfo->getDate().get(lsst::daf::base::DateTime::MJD);
    double latitude = visitInfo->getObservatory().getLatitude();
    _lstObs = visitInfo->getEra();
    _hourAngle = visitInfo->getBoresightHourAngle();

[timj]

ERA can come directly from knowing the time and observatory location so there doesn't need to be a header for that. All these visitInfo items are filled in from the astro_metadata_translator ObservationInfo class.

I've been pushing airmass, and hour angle corresponding to the DATE-OBS and DATE-END. In AuxTel I've suggested AMSTART, AMEND, HASTART, HAEND for these values. Observatory latitude comes from OBSGEO-[XYZ] headers.

[jchiang87]

@boutigny I'm assuming that visitInfo object you refer to in your thread excerpt would be constructed via code like this:

import lsst.daf.persistence as dp
butler = dp.Butler('data_repo')
visit_info = butler.get('raw_visitInfo', dataId=dict(visit=161899, raftName='R22', detectorName='S11'))

in which case the header keywords that need to be set in the PHDU of the raw files are the ones accessed here: https://github.com/lsst/afw/blob/master/src/image/VisitInfo.cc#L317

[timj]

@jchiang87 those headers you link to are added by afw on output and are not headers that are required for raw data. VisitInfo raw data headers are created by the astro_metadata_translator's ObservationInfo parsing the raw header and converting to VisitInfo in obs_base.

[jchiang87]

So how do we ensure that the info that are needed for jointcal are in the raw headers with the right keywords. Dominique posted access via a VisitInfo object, so I assumed that since he was worried about raw files that the raw_visitInfo data product would need to serve up that info. As far as I can tell, using the obs_lsst mapper code as-is, this implies those keywords need to be set in the raw headers, i.e., running the code snippet I posted results in a visit info object that has the data that Dominique's example points to.

[timj]

In legacy obs packages (of which obs_lsst is) the VisitInfo is populated using some defaults from obs_base. See LsstCamMakeRawVisitInfo -- that only sets dark time and date at the moment from what I can see. Once I convert it to the ObservationInfo scheme then everything will be like obs_decam/cfht/subaru.

AFW uses some long keywords that would be best avoided in raw data. In my earlier reply I indicated my preference for the headers that should be used to get this information. If you have headers anywhere for these quantities then the translator will be able to handle it (as I say, do not worry about ERA).

[boutigny]

In jointcal these quantities are accessed through visitInfo in https://github.com/lsst/jointcal/blob/master/src/CcdImage.cc#L110
In my early tests, following Robert's guidance, I modified https://github.com/lsst/obs_lsst/blob/master/python/lsst/obs/lsst/lsstCamMapper.py#L41 to pass these values to visitinfo from the rawheaders.
So if I understand correctly this whole mechanism is changing ?

[timj]

Yes. That mechanism is old and has changed. See https://github.com/lsst/obs_subaru/blob/master/python/lsst/obs/hsc/makeHscRawVisitInfo.py

I'm doing all this on a ticket branch for obs_lsst though since I'm still finishing up the new translators.

[cwwalter]

After all the work on obs_lsst by @timj and @jchiang87 etc. Can this be closed with the merging of PR #194? Or are there other outstanding headers we may not yet be including?

[cwwalter]

@jchiang87 Are we in good shape for this now? There have been several sets added (e.g. in #194).

[jchiang87]

This issue is very old and refers to metadata handling that has changed at least a couple times since it was first posted. I would recommend closing this and making a new issue that addresses the current keyword issues. I can make a new issue shortly.

[cwwalter]

OK closing.