Add support for time-dependent photometry to calwf3

[mackjenn]

**** EDITED January 6th, 2020 ****
This work is HIGH priority for WFC3, as we intend to present these new results at the March 17th workshop 'Accurate Flux Calibration for 21st Century Astrophysics'
Ideally, we would like to test a dummy version of the new file with calwf3 by the end of January.

---

To: Michele De La Pena mdelapena@stsci.edu
From: Jennifer Mack mack@stsci.edu
Cc: Annalisa Calamida calamida@stsci.edu, Sylvia Baggett sbaggett@stsci.edu
Date: Tuesday, October 1, 2019 at 5:59 PM

The WFC3 team is working on an early 2020 delivery which would require calwf3 to interpret time-dependent photometry keywords from the IMPHTTAB reference file and write them to the image headers.

ACS already does this via their IMPHTTAB, which has different columns for various epochs. These define the endpoints with which calacs performs a linear interpolation of the sensitivity based on observation date .

For UVIS, the PHOTMODE values in the image header apply to all epochs: PHOTMODE= ‘WFC3 UVIS1 F475W' and with the new software would need to look something more like the ACS syntax: PHOTMODE= 'ACS WFC1 F475W MJD#54113.9559'

Does calwf3 already have the ability to handle time-dependent sensitivity (eg. is it just commented out), or do you think this is code that will need to be developed? If so, does it only support a linear interpolation, or could one apply a higher order fit?

One difference from ACS is that our IMPHTTAB contains chip-specific sensitivity values, so we have more extensions than usual. For example, below compares ACS/WFC and WFC3/UVIS:

EXT# FITSNAME FILENAME EXTVE DIMENS BITPI OBJECT
0 2391649pj_imp 2391649pj_imp.fits 8
1 BINTABLE PHOTFLAM 1 13Fx780R
2 BINTABLE PHOTPLAM 1 13Fx780R
3 BINTABLE PHOTBW 1 13Fx780R

EXT# FITSNAME FILENAME EXTVE DIMENS BITPI OBJECT
0 0bi2206ti_imp 0bi2206ti_imp.fits 16
1 BINTABLE PHOTFLAM 1 5Fx256R
2 BINTABLE PHOTPLAM 1 5Fx256R
3 BINTABLE PHOTBW 1 5Fx256R
4 BINTABLE PHTFLAM1 1 5Fx256R
5 BINTABLE PHTFLAM2 1 5Fx256R

The timing of this will be important, as we would like to deliver the solutions in early 2020 and we would need to be able to do initial testing in late-2019.

[mackjenn]

From: Michele De La Pena mdelapena@stsci.edu @mdlpstsci

I gave calwf3 a cursory examination. Calwf3 does not take advantage of the time-dependent sensitivity, but the time-dependent code is in a standalone module which was originally designed for use by ACS, STIS, and WFC3. I admit that I will need to look more closely at what is going on in calwf3 and this module to become familiar with its functionality. I will also note for you the code currently only performs a linear interpolation.

I discussed this issue with Warren as he developed much of the original code, and he believes that there should not be much work to have calwf3 exploit the existing time-dependent sensitivity module. I would like to perform a quick experiment just to test “integrating” an updated IMPHTTAB table with calwf3 to get a better feeling what work needs to be done myself. If you have an updated table that I could use for experiments, it would be great. The table does not have to be the real thing, it just needs to have the “time-dependent” aspects incorporated.

Having a higher-order fit accommodated in the time-dependent sensitivity code would involve considerable time. The issue here is that the fit needs to be done over multiple parameters simultaneously (to my understanding). If a higher-order fit seems more appropriate, it could be worth considering having smaller time intervals so that any “fit” within an interval is well approximated by a linear fit.

I suggest opening Git Issue for HSTCAL/WFC3 where we could include this conversation.

[mdlpstsci]

@mackjenn Please provide a "test" IMPHTTAB that can be used for experimentation.

[pllim]

Maybe @sosey remembers something about this. She did a lot of work implementing the IMPHTTAB for WFC3 back in the days.

[mackjenn]

Thanks, Michele. We will provide a dummy IMPHTTAB for testing by the end of October.

[Vb2341]

I have created a time dependent dummy imphttab, and here's the summary:

I started with the current imphttab, which has 5 extensions, each containing a table. Those extensions are the PHOTFLAM (inverse sensitivity), PHOTPLAM (pivot wavelength), PHOTBW (bandwidth), PHTFLAM1 (Chip 1 sensitivity), PHTFLAM2 (Chip 2 sensitivity).

The time dependence only appears in the PHOTFLAM, PHTFLAM1, and PHTFLAM2 extensions, as the pivot wavelength and bandwidth are not affected due to change in sensitivity. I can see this being an annoyance to deal with programatically, as now the extensions would have different numbers of rows/obsmodes, so if it is a pain, I will add the extra rows/columns (described below) to the PHOTPLAM and PHOTBW extensions, just with constant values for any time dependent parts.

The time dependence is contained in the extra columns, containing the substring "MJD#" in the column name, e.g. in the PHOTFLAM extension, there are 5 extra columns with names in the form "PHOTFLAM_MJD#XXXXX", "PHOTFLAM_MJD#YYYYY", etc. where XXXXX, YYYYY, etc are the MJDS which the values in those columns are sampled at. The values of the dates are contained in the column names, as well as header keywords MJD0,MJD1,...MJD4.

However, not all of the rows feature values that change across those 5 columns, as some of the rows in the tables contain OBSMODEs that are NOT time dependent. That is to say the row for the OBSMODE wfc3,uvis1,f814w would have the same value in the 5 PHOTFLAM_MJD#XXXXX columns, whereas the row containing the OBSMODE wfc3,uvis1,f814w,mjd would contain the time dependent values in those 5 columns.

Lastly, for historical reasons, for every obsmode in the current imphttab (not the new one I've made) there was a duplicate obsmode with the cal, key added on the end, i.e. there is a row for the obsmode wfc3,uvis1,f814w, and another row wfc3,uvis1,f814w,cal. I think the CAL mode used to be used for some correction, but is not used now. I left it in there to make sure stuff didn't break, but I did NOT make rows for OBSMODEs such as wfc3,uvis1,f814w,cal,mjd, as that just adds even more redundant information. If calwf3 was looking at the cal modes to read the values, then the software should be changed as to NOT look for those obsmodes (as it'll need to be changed anyway to look for the MJD rows/OBSMODES, I hope this woun't be tough).

The dummy file is located on central store at the following path: /grp/hst/wfc3s/vbajaj/imphttab_mjd/test_imp.fits, let me know if there are any issues with this. I know this post is long winded, and is probably way easier to explain while looking at the file, so let me know if there's any confusion.

[stscicrawford]

As there are column changes, this will likely require a CRDS update. Please open up tickets in the CRDS project for the necessary changes. Please make sure to include @eslavich and me on the ticket

[mackjenn]

This work is very high priority for WFC3, as we intend to present the new calibration at the workshop 'Accurate Flux Calibration for 21st Century Astrophysics' on March 17, 2020.

Ideally, we would like to test the new calwf3 with a dummy version of the modified reference file before the end of January, since significant code changes are required and we may need to iterate with the developer. Thanks!!

[mdlpstsci]

This is just to let folks know this morning I have been examining the dummy imphttab file and calwf3 in order to clarify how many changes and how much time will be needed to implement this change.

[mdlpstsci]

Sent via email:
I have been examining your new WFC3 IMPHTTAB test file in conjunction with the ACS
time-dependent table and the code written to support this reference file. After gaining
a bit of familiarity with the file (in particular the PHOTFLAM table) and discussing with
Warren who implemented the original source code in support of the IMPHTTAB, your new table
format does not conform to the structure established by ACS and the source. It is
understood that your file has additional extensions, so your file cannot be the same as
the ACS case, but the tables do need to follow the existing format.

I can see the latest ACS WFC IMPHTTAB was created by Jenna Ryon using create_table()
which is a routine in the Git Repo "spacetelescope/reftools/reftools/mkimphttab.py. I do
not know how helpful this utility would be for you.

I think we should get together to discuss this issue as soon as possible.

[mackjenn]

Thank you for the very informative meeting! We will make the necessary changes and get back to you soon.

[mdlpstsci]

Varun created a new reference file for testing.

I was able to create a dummy IMPHTTAB that matches the format of the other time dependent files, such as those for ACS or STIS. The only real departure that I have from those is that they use name PHOTFLAM1 for the time dependent column (or PHOTBW1 or PHOTPLAM1), but we already use the name "PHTFLAM1" for one of the extensions, so all of my time dependent column names are the name of the normal column + 'T', i.e. PHOTFLAMT, PHOTBWT, PHTFLAM1T. I did this in hopes that it would avoid confusion. If this is also an issue for the code, let me know. I've attached the file here. (I have the FITS file).

[mdlpstsci]

Sent to Varun (et al) on 03 March 2020.

I have analyzed the new imphttab file and the code which accesses
this file. I have determined there are some items which need to be
changed in order for this file to adhere to the stringent format
required by the code.

---

TO CHANGE

1. The PARNUM keyword in the Primary header needs to be set to 1.

2. All of the extensions should have the EXTNAMEs and EXTVER values
   set explicitly. For WFC3 these names must be in order:
   PHOTFLAM, PHOTPLAM, PHOTBW, PHTFLAM1, and PHTFLAM2
   All the EXTVER values should be 1.

3. The suffix of "T" will not work for the column names. I can
   appreciate your wanting to avoid possible confusion as this
   entire data structure is complex, but the column names are constructed
   on-the-fly and indexed by number.

4. As a consequence of Item (3) the DATACOL values also need to be
   modified.

FYI
5) If you want extrapolation turned on, you need to add the EXTRAP
keyword to the Primary header of the reference file with a value
of True.

6. All the tables must have the same rows, or more specifically, the
   same "obsmode" strings available.

7. I recall you wanted to stop adding "cal" to the PHOTMODE keyword
   so that you could remove the "cal" entries in the imphttab. The
   calwf3 currently adds "cal" for UVIS exposures to the PHOTMODE
   string value. I am trying to find out the rationale behind the use
   of "cal" and if it still serves a purpose.

8. I believe I only have to make a minor modification to calwf3 to
   accommodate the parameterization for MJD.

[mackjenn]

Hi Michele,
We are concerned with the code appending a '1' to signify time-dependence and wondered if this can be changed to a character instead of a number (eg. 'T'). Otherwise this is going to be extremely confusing to our users, since we will have:
PHOTFLAM, PHOTFLAM1, PHTFLAM1, PHTFLAM11, PHTFLAM2, PHTFLAM21.
If it is a significant amount of work to make this change, can you give an estimate to the time involved and we can run it by Sylvia who can discuss with the Mission Office to determine if it's worth the extra work. Thanks!

[mdlpstsci]

Just confirming that I saw your email. I have been dealing with other issues and not had time to examin the situation and decide how to handle the issue of unfortunate (but understandable) naming convention.

[mdlpstsci]

Abridged version of an email send to WFC3 team members.

The request to modify the IMPHTTAB structure and terminology cannot be accommodated.

Please find attached an out-of-date document, "Image Photometry Table (IMPHTTAB) Reference File Format" written in 2011. This is a Technical Science Report which explains the philosophy and construction of the IMPHTTAB reference file at the time. The IMPHTTAB reference file has not only a very specific (albeit complex) structure, but it also has a specificsemantic interpretation corresponding to the structure which is baked into CALXXX. A program, mkimphttab.py (Git spacetelescope/reftools) was provided which will construct the IMPHTTAB for you. However, I am guessing this routine has not been updated to accommodate the additional extensions of the WFC3.

The file was designed to be flexible for different instruments and
instrument teams. The characteristics which are parameterized are fully
defined in the file and not known to the software (admittedly there is one
very small change). Everything works properly as the software obeys
the rules set up by the structure and information in the file by design.
The most logical and generic way to track different parameterizations is
with a numeric value (in this case a suffix), particularly if there are
multiple parameterizations. This is where you want to use "T" which
breaks the flexibility and general nature of the code.

[mdlpstsci]

@mackjenn @Vb2341 Forging ahead, I have tested the updated version of CALWF3 to make sure the new file is read and processing ends successfully (if only from a mechanical perspective). I did find a bug in very old code as the code has never been used where there are multiple Imsets and parameterization. Can you please provide just a few datasets which you know would access different rows of the new table for further testing by me? Thanks.

[Vb2341]

@mdlpstsci By multiple imsets, does that mean multiple chips? If so any full frame UVIS image should do. Here are a couple of examples:

ICAU96020
ID5E15010
ICWZ02021

For a given filter, those should access both the UVIS1 and UVIS2 rows.

[mdlpstsci]

@Vb2341 Yes and thank you for the suggested datasets.

[mdlpstsci]

WFC3 delivered what should be a nearly final version to me on 24 July 2020. I need to validate it the best I can (as was done for the preliminary version of the file), and use it for testing.

[mdlpstsci]

@Vb2341 @mackjenn
I have examined the new IMPHTTAB file, new_time_dependent_imp.fits,
and I see that you have modified it to conform to the structure defined
for support of HST instruments. I have also successfully processed
a dataset (icau096020) using this updated IMPHTTAB file, in conjunction
with the modified version of CALWF3. By "successfully" I mean that I
have verified the software computed the correct values for the
photometric variables (PHOTFLAM, PHOTPLAM, PHOTBW, PHTFLAM1, and
PHTFLAM2) based upon MJD of the dataset and the "obsmode"
string (e.g., "wfc3,uvis1,f275w,MJD#).

The updated CALWF3 software is available on my branch,
issue451-Tdep_wfc3_photometry, in GitHub. If you like, I can set
up the software for you (plhstins1?) so you can process datasets
as you deem appropriate. This is similar to what I have done
for the WFC3 folks working on the new CTE correction.

[protonchain]

@mdlpstsci This is Harish, I'm planning to help out with the testing of the software on various datasets. I was wondering if it would be possible to set up an executable of the updated CALWF3 on plhstins1.

Thanks!

[mdlpstsci]

@protonchain I can do this for you later today or tomorrow in the worst case!

[protonchain]

@mdlpstsci Sounds great, thank you!

[mdlpstsci]

I have compiled calwf3 containing the time-dependent photometry update on plhstins1 (Linux). The executable is located in

/home/mdelapena/repos_tdep/hstcal/bin.Linux-2.6.32-754.31.1.el6.x86_64-x86_64-with-redhat-6.10-Santiago

Just to make sure all the paths are resolved and shared objects found, I advise doing the following:

1. If you are not running a bash shell, invoke a bash shell by typing “bash” (without the quotes).
2. Type, export PATH=/home/mdelapena/miniconda3/bin:$PATH
3. Type, source activate wfc3_tdep_2020
4. Type, /home/mdelapena/repos_tdep/hstcal/bin.Linux-2.6.32-754.31.1.el6.x86_64-x86_64-with-redhat-6.10-Santiago/calwf3.e your_test_data_here.fits

This version of the code has the Git commit hash 75af6e6. This is printed to stdout at the very beginning of CALWF3 processing.

[mdlpstsci]

Git #518 which resolves this issue was closed on 20 September 2020.