Merge pull request #119 from Keck-DataReductionPipelines/develop

v6.0 release
Keck-DataReductionPipelines · Apr 17, 2022 · de57c46 · de57c46
2 parents bc0e44f + 53dcd12
commit de57c46
Show file tree

Hide file tree

Showing 9 changed files with 143 additions and 60 deletions.
diff --git a/INSTALLPROBLEMS.md b/INSTALLPROBLEMS.md
@@ -2,6 +2,23 @@
 
 This file contains a bunch of common pipeline installation problems and their solutions.
 
+## Installing on Apple M1 ARM Chips
+
+There is an issue if CFITSIO is not compiled to be the same architecture as the DRP. CFITSIO should be compiled to be x86_64.
+
+If installing CFITSIO from Macports:
+
+```
+arch -x86_64 sudo port install libgcc-devel
+arch -x86_64 sudo port install cfitsio
+```
+
+In the Makefile for the OSIRIS DRP, update the CFLAGS variable:
+
+```
+CFLAGS += -arch x86_64
+```
+
 ## Wrong Architecture Errors
 
 If you get an error like this:
@@ -57,3 +74,13 @@ You can check what is in your PATH by entering the following command:
 ```
 
 If the IDL executable is in your PATH and you are running this on a MAC then there may be a compatibility issue between your versions of XQuarts, IDL, and ENVI. This problem and the solution to it is documented on the Harris Geospatial website [here.] (http://www.harrisgeospatial.com/Home/NewsUpdates/TabId/170/ArtMID/735/ArticleID/14944/XQuartz-2710-is-Not-Compatible-with-ENVI-531-and-IDL-851.aspx)
+
+## Other startup errors
+
+If the pipline complains that ``csh`` or ``xterm`` are not installed, you will need to install them. You don't need to use cshell, but the startup script will automatically call it. 
+
+If you are unable to get ``run_odrp`` script to work, you can try to start the pipeline manually using (you will still need to setup your environment variables correctly):
+
+```
+idl -IDL_STARTUP ${OSIRIS_BACKBONE_DIR}/drpStartup.pro ${OSIRIS_BACKBONE_DIR}/osiris_drp_backbone_startup.pro
+```
diff --git a/OSIRIS_Manual_v6.pdf b/OSIRIS_Manual_v6.pdf
diff --git a/README.md b/README.md
@@ -1,12 +1,18 @@
-# Keck OSIRIS Data Reduction Pipeline
+# Keck OSIRIS Spectroscopic Data Reduction Pipeline
 
-* [Release Notes for v5.0](#release-notes-for-v5.0)
+* [Release Notes for v6.0](#release-notes-for-v6.0)
 * [Important Runtime Notes](#important-runtime-notes)
 * [Installation](#installation)
 * [Running the Pipeline](#running-the-pipeline)
 * [Testing the Pipeline](#testing-the-pipeline)
 * [Citing the Pipeline](#citing-the-pipeline)
 
+## Release Notes for v6.0
+**2022-04-12**
+- Update to the manual including: discussion of new imager, new tables of sensitivities for imager and spectrograph, updates on how to observe with the TRICK NIR TT sensor, discussion of the new exposure time calculator, new discussion of the OSIRIS imager reduction pipeline KAI.
+- The OSIRIS imager pipeline is now available as KAI in a seperate repository: [https://github.com/Keck-DataReductionPipelines/KAI](https://github.com/Keck-DataReductionPipelines/KAI)
+- Updated installation instructions for Apple M1 architecture.
+
 
 ## Release Notes for v5.0
 **2021-02-22**
@@ -21,48 +27,9 @@
 - Made a slight update to IDL_astro routine xy2ad.pro to handle WCS in OSIMG images.
 - Other minor updates
 
+**Previous Release Notes**
 
-## Release Notes for v4.2.0
-**2018-05-07**
-Major Updates
-- Derived new wavelength solution for OSIRIS in March 2018, which was required because OSIRIS was opened. All users are recommended to use this version, especially those with data post March 2018.
-- **Updated manual** - new installation instructions, wavelength solution, bad pixel mask info, updated information post-2016 detector upgrade are ongoing. Download [here](OSIRIS_Manual_v4.2.pdf)
-
-## Release Notes for v4.1.0
-**2017-11-07**
-Major Updates
-- Includes a new wavelength solution for data after May 2017. A shift in the wavelength solution (on average about 2.8 Angstroms offset) in May 2017 required a re-derivation of the solution. The new solution has an average offset between the observed and vacuum wavelength of OH lines of 0.07 +- 0.06 Angstroms in Kn3 35 mas.
-- A preliminary bad pixel mask is available for data taken after 2016 (new spectrograph detector). The mask was computed from a series of darks. There is both a bad pixel mask of hot pixels (pixels with permanently elevated value) as well as a dead pixel mask (pixels with permanently low values). This mask meant to be used as extension 2 in the raw fits files. Currently, the mask is not automatically applied by Keck. To apply it, use the following command in the raw spectra directory once the pipeline is installed:
-```
-apply_mask.py *.fits
-```
-NOTE: this requires python installed with ``numpy`` and ``astropy`` packages. Tests show that using the bad pixel mask improves the SNR by about 10%.
-
-- A new keyword is available in the Scaled Sky Subtraction module called ``scale_fitted_lines_only``. To turn on the new behavior, the keyword should be set to YES  and the ``Scale_K_Continuum`` should be set to NO:
-```
-scale_fitted_lines_only='YES'
-Scale_K_Continuum='NO'
-```
- This keyword will only scale only OH lines, not the rest of the spectrum as well. This setting greatly improves sky subtraction for the case where the science target fills the lenslets and there are no true sky locations. It may also help in other cases. Users are encouraged to try this option if they see large residuals in sky subtraction, or if the residual continuum is problematic.
-
-- The cosmic ray module is now automatically turned off for all data with the new detector (see reasoning below). Cosmic rays represent about 1% of the bad pixels in a typical 15 minute exposure -- the majority are static bad pixels that should now be accounted for by the bad pixel mask.
-
-## Release Notes for v4.0.0
-**2017-01-23**
-
-Major updates:
-- Updates to the code in order to run the pipeline for the new detector (2016 data and newer).
-- Installation has now been simplified (see below for install directions). Bash scripts have been included for those who would like to use bash shell.
-- Test framework is now available to run tests of the pipeline (requires pytest module in python, see README in ''tests'' directory)
-- Optimized algorithms for the construction of data cubes
-- qlook2, odrfgui, and oopgui are now also included in the repository
-
-Minor Updates:
-- WCS bugs have been fixed.
-- qlook2 fixes
-  - units bug is fixed
-  - fix initial autoscale to imager display
-  - clean up startup scripts
+For older release notes see: RELEASE_NOTES.md
 
 ## Important Runtime Notes
 
@@ -103,6 +70,8 @@ To install and run the OSIRIS DRP, you will need the following:
 - A copy of the compiled library cfitsio
 - A working installation of IDL 7 or IDL 8 (the IDL binary directory should be in your ``PATH`` environment variable)
 - Python dependencies (optional, for testing): pytest, astropy
+- If using a computer with Apple M1 ARM chips, see [INSTALLPROBLEMS.md](https://github.com/Keck-DataReductionPipelines/OsirisDRP/blob/master/INSTALLPROBLEMS.md) for a workaround.
+- ODRFGUI: Java version 17 (newer versions of Java are likely to run into issues when running the GUI)
 
 
 ### Installing from source
@@ -175,21 +144,8 @@ setenv PATH ${PATH}:/Applications/exelis/idl/bin
 To run the pipeline, use ``run_odrp``. If you don't want the pipeline
 to open a new xterm window, call ``run_odrp -n``.
 
-Please check out the OSIRIS pipeline manual: <http://www2.keck.hawaii.edu/inst/osiris/OSIRIS_Manual_v2.3.pdf>
-
-### Testing the Pipeline
-
-To run the suite of tests on the pipeline, and you have ``pytest`` and ``astropy`` in your python environment:
-
-```
-make test
-```
-
-The first time you run the tests, data will be downloaded so it will take longer. If the tests pass, your pipeline is installed properly. You will see something like the following if the tests pass:
+Please check out the OSIRIS pipeline manual: OSIRIS_Manual_v5.pdf in this directory.
 
-```
-======================== 2 passed, 2 skipped in 41.77 seconds ===================
-```
 
 ### OOPGUI &  ODRFGUI settings
 
@@ -200,7 +156,7 @@ To set the default directories for the guis, you can edit the following two file
 
 ### Troubleshooting
 
-If you run into problems, please re-read this [README.md](https://github.com/Keck-DataReductionPipelines/OsirisDRP), then read [INSTALLPROBLEMS.md](https://github.com/Keck-DataReductionPipelines/OsirisDRP/blob/master/INSTALLPROBLEMS.md) for some common installation problems.
+If you run into problems, please re-read this [README.md](https://github.com/Keck-DataReductionPipelines/OsirisDRP), then read [INSTALLPROBLEMS.md](https://github.com/Keck-DataReductionPipelines/OsirisDRP/blob/master/INSTALLPROBLEMS.md) for some common installation problems. Please post an issue on the issue page if you have problems not addressed in the above documents or in the manual. 
 
 OSIRIS DRP Project Contributors
 ============================

diff --git a/RELEASE_NOTES.md b/RELEASE_NOTES.md
@@ -0,0 +1,62 @@
+## Release Notes for v6.0
+**2022-04-12**
+- Update to the manual including: discussion of new imager, new tables of sensitivities for imager and spectrograph, updates on how to observe with the TRICK NIR TT sensor, discussion of the new exposure time calculator, new discussion of the OSIRIS imager reduction pipeline KAI.
+- The OSIRIS imager pipeline is now available as KAI in a seperate repository: [https://github.com/Keck-DataReductionPipelines/KAI](https://github.com/Keck-DataReductionPipelines/KAI)
+- Updated installation instructions for Apple M1 architecture.
+
+
+## Release Notes for v5.0
+**2021-02-22**
+- New wavelength solution for 2021. OSIRIS had to be opened for servicing in Dec. 2020, so new arc scans were taken in Jan 2021 and new wavelength solutions were created. Tests show that the average wavelength shift at Kn3 35 mas to be about -0.04+-0.07 Angstroms (based on comparisons with OH skylines). At Kn3 50 mas, the shift is on average: 0.38+-0.06 Angstroms. 
+
+## Release Notes for v5.0beta
+**2020-09-22**
+- New wavelength solution for 2019 & 2020. Unlike data before 2019, there appears to be larger residual offsets in the wavelength solution between different plate scales. The smallest shift is at 50 mas and the grows larger for smaller plate scales. At Kn3 35 mas, the offset is about 0.3 Angstrom based on comparisons with OH sky lines. 
+- Handle imager upgrade pixel units (DN) instead of DN/s
+- The FITS files from the imager upgrade were flipped such that the images were not in an astronomical orientation.
+- QL2 will now flip IMAGER images about the x-axis (IDL-> im=reverse(im,2)) for upgraded images only.  SPEC and DRP cubes are NOT flipped.
+- Made a slight update to IDL_astro routine xy2ad.pro to handle WCS in OSIMG images.
+- Other minor updates
+
+
+## Release Notes for v4.2.0
+**2018-05-07**
+Major Updates
+- Derived new wavelength solution for OSIRIS in March 2018, which was required because OSIRIS was opened. All users are recommended to use this version, especially those with data post March 2018.
+- **Updated manual** - new installation instructions, wavelength solution, bad pixel mask info, updated information post-2016 detector upgrade are ongoing. Download [here](OSIRIS_Manual_v4.2.pdf)
+
+## Release Notes for v4.1.0
+**2017-11-07**
+Major Updates
+- Includes a new wavelength solution for data after May 2017. A shift in the wavelength solution (on average about 2.8 Angstroms offset) in May 2017 required a re-derivation of the solution. The new solution has an average offset between the observed and vacuum wavelength of OH lines of 0.07 +- 0.06 Angstroms in Kn3 35 mas.
+- A preliminary bad pixel mask is available for data taken after 2016 (new spectrograph detector). The mask was computed from a series of darks. There is both a bad pixel mask of hot pixels (pixels with permanently elevated value) as well as a dead pixel mask (pixels with permanently low values). This mask meant to be used as extension 2 in the raw fits files. Currently, the mask is not automatically applied by Keck. To apply it, use the following command in the raw spectra directory once the pipeline is installed:
+```
+apply_mask.py *.fits
+```
+NOTE: this requires python installed with ``numpy`` and ``astropy`` packages. Tests show that using the bad pixel mask improves the SNR by about 10%.
+
+- A new keyword is available in the Scaled Sky Subtraction module called ``scale_fitted_lines_only``. To turn on the new behavior, the keyword should be set to YES  and the ``Scale_K_Continuum`` should be set to NO:
+```
+scale_fitted_lines_only='YES'
+Scale_K_Continuum='NO'
+```
+ This keyword will only scale only OH lines, not the rest of the spectrum as well. This setting greatly improves sky subtraction for the case where the science target fills the lenslets and there are no true sky locations. It may also help in other cases. Users are encouraged to try this option if they see large residuals in sky subtraction, or if the residual continuum is problematic.
+
+- The cosmic ray module is now automatically turned off for all data with the new detector (see reasoning below). Cosmic rays represent about 1% of the bad pixels in a typical 15 minute exposure -- the majority are static bad pixels that should now be accounted for by the bad pixel mask.
+
+## Release Notes for v4.0.0
+**2017-01-23**
+
+Major updates:
+- Updates to the code in order to run the pipeline for the new detector (2016 data and newer).
+- Installation has now been simplified (see below for install directions). Bash scripts have been included for those who would like to use bash shell.
+- Test framework is now available to run tests of the pipeline (requires pytest module in python, see README in ''tests'' directory)
+- Optimized algorithms for the construction of data cubes
+- qlook2, odrfgui, and oopgui are now also included in the repository
+
+Minor Updates:
+- WCS bugs have been fixed.
+- qlook2 fixes
+  - units bug is fixed
+  - fix initial autoscale to imager display
+  - clean up startup scripts
diff --git a/backbone/drpStartup.pro b/backbone/drpStartup.pro
@@ -28,7 +28,7 @@ IF (readcol_path EQ '') THEN EXIT, /NO_CONFIRM
 
 .compile skysclim.pro
 .compile strn.pro
-.compile buie_avgclip.pro
+;.compile buie_avgclip.pro
 .compile strnumber.pro
 .compile ICG_LIB.pro
 .compile break_path.pro

diff --git a/backbone/version.txt b/backbone/version.txt
@@ -1 +1 @@
-5.0beta
+6.0
diff --git a/data/Wavelength_Shift_Time_Dependence.png b/data/Wavelength_Shift_Time_Dependence.png
diff --git a/data/wavelength_solution_dates.txt b/data/wavelength_solution_dates.txt
@@ -0,0 +1,11 @@
+date		ddate
+2005-02-22	2005.1429              
+2006-02-23	2006.1450
+2009-10-05	2009.7590
+2012-01-04	2012.0067
+2016-01-01	2015.9985
+2017-05-09	2017.3510
+2018-03-16	2018.2025
+2019-05-01	2019.3277
+2020-01-01	2019.9985
+2020-12-16	2020.9568
diff --git a/data/wavelength_solution_shift.txt b/data/wavelength_solution_shift.txt
@@ -0,0 +1,27 @@
+Date		DecDate		Mean		Sigma			Filter	Scale
+2006-07-01	2006.4954	-0.19072390	0.075718629		Kn3		35
+2007-07-20	2007.5468	0.088797292	0.14183580		Kn3		35
+2008-05-16	2008.3709	-0.36417465	0.10578299		Kn3		35
+2009-05-07	2009.3455	-0.0061323681	0.06116898		Kn3		35
+2010-05-05	2010.3394	-0.00501985	0.048214412		Kn3		35
+2012-05-10	2012.3544	0.24504756	0.065833827		Kn3		35
+2013-05-16	2013.3712	0.22	0.12	Kn3	20
+2014-05-18	2014.3767	-0.28	0.11	Kn3	35
+2015-07-21	2015.5495	-0.6534200	0.42660178		Kn3		35
+2016-04-18	2016.2942	0.15973073	0.092983175		Jn2		35
+2016-04-21	2016.3034	-0.09		0.39			Kbb		35
+2017-05-17	2017.3747	0.07	0.06	Kn3		35
+2017-11-03	2017.8398	-0.88	0.04	Kbb	20
+2018-07-31	2018.5788	0.04	0.05	Kn3		35
+2019-05-11	2019.3562	0.33	0.06	Kn3		35
+2019-08-26	2019.6491	-0.01	0.05	Kn3		50
+2020-07-23	2020.5581	0.23	0.07	Kn3		35
+2020-07-29	2020.5747	0.26	0.03	Kbb		35
+2020-07-30	2020.5777	0.72	0.11	Kbb		20
+2020-08-13	2020.6157	0.71	0.08	Kn3		20
+2021-02-10	2021.1102	-0.04	0.07	Kn3		35
+2021-02-10	2021.1102	0.38	0.06	Kn3		50
+2021-05-07	2021.3471	0.01	0.07	Kn3 	35
+2021-05-16	2021.3719	0.41	0.08	Kn3		20
+2021-07-10	2021.5191	-0.03	0.07	Kn3		35
+2021-07-19	2021.5465	0.42	0.04	Kbb		20