DM-39649: Add default LATISS transmission_detector and transmission_optics. #13

erykoff · 2023-07-11T17:03:23Z

No description provided.

This was taken from https://srs.slac.stanford.edu/DataCatalog/dataset.jsp?dataset=40352803&experiment=LSST-CAMERA

timj · 2023-07-11T17:07:04Z

latiss/transmission_sensor/README.md

+
+This was then converted to the calib format by running the script `python $OBS_LSST_DIR/python/lsst/obs/lsst/script/rewrite_latiss_qe_file.py`.
+
+Both the original FITS and output ecsv files have been committed to this repo.


Maybe confirm that it's the ECSV version that is the one used by the calibration system?

Are you worried the other files there might be picked up? Should they go in a different directory? But the calibration ecsv file is the one that is used, and it is the only one with the correct name. (On jira I have a script that shows that the files are working as intended).

Is there a downside to an explicit comment? It might be that it would be best to put the original source files somewhere else. It does occur to me as well that it might be nice to add provenance to the ECSV file indicating where the data came from.

Sorry, I don't quite understand what you're suggesting.

Which part?

Adding a keyword is a good idea, although we've already been inconsistent on this. I think my suggestion here is going to be to choose any reasonable key name to add the source file name to the newly added ECSV files. As this ticket already has edits on the LATISS transmission_filter files, those should be updated to that same keyword (it's just SOURCE now).

The FITS file is safe from being picked up as we restrict the inputs in this package to be one of {.ecsv, .yaml, json}, so there's no reason to move them out of the way now. I think moving them to a _src directory makes sense, but I don't have a clear idea of where that directory should live (do we have one top level _src containing all of our sources, do we have one for each instrument, or do we have one for each calibration type), so I propose adding a "clean up" ticket that can handle that decision, do the moves, and enforce whatever keyword choices onto the existing files without making this ticket any bigger.

So should I just add a SOURCE key to each of these ecsv files? That makes sense to me.

Okay. At some point we should make the rewrite script do that.

Yup, that script is on this ticket (in obs_lsst), so that's where I was going to put it. It's simply a question of the name of the key.

SOURCE works for me.

czwa

A few questions throughout, and a request to add some header keyword denoting the sources to the files that will be ingested as curated calibrations.

czwa · 2023-07-11T21:42:52Z

latiss/transmission_filter/sdssg_65mm~empty/20221005T000000.ecsv

@@ -19,7 +19,7 @@
 # - __serialized_columns__:
 #     throughput:
 #       __class__: astropy.units.quantity.Quantity
-#       unit: !astropy.units.Unit {unit: ''}
+#       unit: !astropy.units.Unit {unit: ""}


Does this remove the "unknown unit" issue, or is this just switching to a consistent quote style?

This removes the unknown unit issue, strangely.

czwa · 2023-07-11T21:48:38Z

latiss/transmission_optics/README.md

+Visual inspection shows that the AuxTel mirror reflectivity at the 7 sampled wavelengths is on average consistent with the Subaru mirror as scanned on 19 Feb, 2020.
+The file "subaru_m1_r_20200219.txt" was retrieved from https://www.naoj.org/Observing/Telescope/Parameters/Reflectivity/subaru_m1_r_20201209.txt on 10 July 2023, and then edited to remove some erroneous spaces that made it so Astropy was unable to read it properly, as well as the second measurements between 950 and 1000 nm.
+
+The final mirror transmission used as the AuxTel baseline is then taken as the Subaru M1 measurement from 19 February, 2020, and adjusted by 3.5% to match the average of the throughput measurements.


Was this fit done for a particular date, or was the full ensemble of AuxTel measurements used to determine the scale factor to match the Subaru measurement?
Are planning to have our own high resolution reflectivity scan in the future? I'm concerned that the mirror degradation between AuxTel and Subaru will be different, as I would expect Subaru to have a higher rate of volcanic dust deposition than AuxTel.

I've updated the text:

The final mirror transmission used as the AuxTel baseline is then taken as the Subaru M1 measurement from 19 February, 2020, and adjusted by 3.5% to match the average of the AuxTel throughput measurements over 3 years of measurements. The chromatic shape of the mirror transmssion recorded in this repo is then consistent with the average AuxTel measurements, but at higher resolution, including the complex shape of the reflectivity of aluminum around 800 nm that are not captured in the AuxTel scans. For calibration purposes, the chromatic shape of the mirror reflectivity is much more important than the absolute throughput which is degenerate with numerous other effects.

And interestingly it does not look like the chromatic shape of the AuxTel mirror reflectivity changed very much over the few years that the scanning has been done.

czwa · 2023-07-11T21:52:12Z

latiss/transmission_optics/README.md

+The final mirror transmission used as the AuxTel baseline is then taken as the Subaru M1 measurement from 19 February, 2020, and adjusted by 3.5% to match the average of the throughput measurements.
+The mirror transmssion recorded in this repo is then consistent with the AuxTel measurements, but at higher resolution, including the rapid changes around 800 nm that are not captured in the AuxTel scans.
+
+Note that this file only contains reflectivity from AuxTel M1, and not from any of the other optical elements.


This is another concern. I think a note on the JIRA ticket that either details how we'll update these in the future or stating that a plan needs to be made (with an additional ticket for that work?) would be good so it's clear if/when these will be updated.

I think the medium-term plan for updating into the future would be to make use of the in-situ measurements of the system throughput using the monochrometer + photodiode + fiber spectrograph. I'm not sure how this is all ticketed and planned, though.

czwa · 2023-07-11T21:52:43Z

latiss/transmission_optics/auxtel_mirror_052923.csv

+4/27/23,85.18,86.05,87.25,88.00,87.55,85.50,91.58
+5/12/23,86.07,87.03,88.20,88.17,87.67,86.13,91.27
+5/18/23,85.68,86.55,87.55,88.33,87.68,85.53,91.53
+5/29/23,85.20,86.63,87.67,86.97,86.87,84.83,90.53


Missing final newline.

czwa · 2023-07-11T21:55:41Z

latiss/transmission_optics/subaru_m1_r_20200219.txt

+384	80.78	81.20	81.46	82.19	81.83
+382	81.51	81.40	81.89	81.95	81.26
+380	81.63	82.02	81.34	80.81	81.44
+


Is this a significant gap, or is it just a result of the different wavelength ranges? The README states that this has already been edited for astropy legibility, so I just wanted to check that there are no nan nan nan nan nan nan rows being populated.

It just skips those rows, but I can delete them because I've already edited the file.

czwa · 2023-07-11T22:21:35Z

latiss/transmission_sensor/README.md

+
+This was then converted to the calib format by running the script `python $OBS_LSST_DIR/python/lsst/obs/lsst/script/rewrite_latiss_qe_file.py`.
+
+Both the original FITS and output ecsv files have been committed to this repo.


Adding a keyword is a good idea, although we've already been inconsistent on this. I think my suggestion here is going to be to choose any reasonable key name to add the source file name to the newly added ECSV files. As this ticket already has edits on the LATISS transmission_filter files, those should be updated to that same keyword (it's just SOURCE now).

The FITS file is safe from being picked up as we restrict the inputs in this package to be one of {.ecsv, .yaml, json}, so there's no reason to move them out of the way now. I think moving them to a _src directory makes sense, but I don't have a clear idea of where that directory should live (do we have one top level _src containing all of our sources, do we have one for each instrument, or do we have one for each calibration type), so I propose adding a "clean up" ticket that can handle that decision, do the moves, and enforce whatever keyword choices onto the existing files without making this ticket any bigger.

czwa · 2023-07-11T22:24:37Z

latiss/transmission_optics/19700101T000000.ecsv

+# - {CALIBDATE: '1970-01-01T00:00:00'}
+# - {INSTRUME: LATISS}
+# - {OBSTYPE: transmission_optics}
+# - {CALIBCLS: lsst.ip.isr.IntermediateOpticsTransmissionCurve}


Pulling from the discussion below: this file also should have a keyword noting the data source.

czwa · 2023-07-11T22:25:19Z

latiss/transmission_sensor/rxx_s00/19700101T000000.ecsv

+# - {INSTRUME: LATISS}
+# - {OBSTYPE: transmission_sensor}
+# - {DETECTOR: 0}
+# - {CALIBCLS: lsst.ip.isr.IntermediateSensorTransmissionCurve}


And a source keyword here, just for completeness.

erykoff added 2 commits July 11, 2023 10:02

Add QE fits file for LATISS sensor.

d7a138e

This was taken from https://srs.slac.stanford.edu/DataCatalog/dataset.jsp?dataset=40352803&experiment=LSST-CAMERA

Add README.md for transmission_sensor processing.

d5eea2e

timj reviewed Jul 11, 2023

View reviewed changes

czwa approved these changes Jul 11, 2023

View reviewed changes

erykoff added 4 commits July 13, 2023 12:04

Add converted transmission_sensor ecsv file.

38cd234

Manually fix transmission units from illegal - to "".

e97b373

Add input data for creating transmission_optics file.

c099bc0

Add default transmission_optics file for LATISS.

2052ec8

erykoff force-pushed the tickets/DM-39649 branch from 9ebdf33 to 2052ec8 Compare July 13, 2023 19:05

erykoff merged commit 5736ae6 into main Jul 14, 2023
3 checks passed

erykoff deleted the tickets/DM-39649 branch July 14, 2023 21:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DM-39649: Add default LATISS transmission_detector and transmission_optics. #13

DM-39649: Add default LATISS transmission_detector and transmission_optics. #13

erykoff commented Jul 11, 2023

timj Jul 11, 2023

erykoff Jul 11, 2023

timj Jul 11, 2023

erykoff Jul 11, 2023

timj Jul 11, 2023

czwa Jul 11, 2023

erykoff Jul 12, 2023

timj Jul 12, 2023

erykoff Jul 12, 2023

czwa Jul 12, 2023

czwa left a comment

czwa Jul 11, 2023

erykoff Jul 12, 2023

czwa Jul 11, 2023

erykoff Jul 12, 2023

czwa Jul 11, 2023

erykoff Jul 12, 2023

czwa Jul 11, 2023

czwa Jul 11, 2023

erykoff Jul 12, 2023

czwa Jul 11, 2023

czwa Jul 11, 2023

czwa Jul 11, 2023


		This was then converted to the calib format by running the script `python $OBS_LSST_DIR/python/lsst/obs/lsst/script/rewrite_latiss_qe_file.py`.

		Both the original FITS and output ecsv files have been committed to this repo.

DM-39649: Add default LATISS transmission_detector and transmission_optics. #13

DM-39649: Add default LATISS transmission_detector and transmission_optics. #13

Conversation

erykoff commented Jul 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

czwa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment