How to run pipeline
If you are just getting started, see How to Install for how to install the IGRINS PLP and set up your raw data.
The IGRINS PLP runs in Python and there is no separate compilation process. Once it is cloned or downloaded and you have installed the required Python dependencies, you should be able to run it. You are supposed run everything within the source directory (this may change eventually). In the source directory, you should see a file called "igr_pipe.py", which is the main executable to run the pipeline. Try to run it like below, and you should see a similar error message.
> python ./igr_pipe.py
usage: igr_pipe.py [-h]
{flat,thar,sky-wvlsol,a0v-ab,stellar-ab,extended-ab,extended-onoff,plot-spec,publish-html}
...
igr_pipe.py: error: too few argumentsAll the IGRINS data is organized by nights which follow the date format YYYYMMDD. To run the pipeline, you need to have a recipe log file for a given night listing observation groups and recipe names to be applied. The recipe log file serves to tell the pipeline what to do for a given night. This is necessary as no recipe information is recorded in the fits header or other logs during the observations. These recipe log files are stored in the directory called "recipe_logs".
> ls recipe_logs
20140524.recipes 20140707.recipes 20140710.recipes 20140713.recipes
20140525.recipes 20140708.recipes 20140711.recipes
20140526.recipes 20140709.recipes 20140712.recipesYou might have been provided a recipe log with your data, or downloaded one from RRISA. You might want to modify it, or if one was not provided need to create your own. See How to prepare recipe logs for instruction to prepare the recipe log file. Here we assume that you already have the recipe log.
To simplify execution of the IGRINS PLP, we provide an example script called "run.sh" in the source directory. For most users, reducing a night of IGRINS data simply consists of you running the script after you have prepared the night's recipe log. This script is set up so that the PLP runs "python igr_pipe.py" for nearly all possible recipes you might encounter in the correct order with the optimal settings. In some circumstances, you might want to create your own modified version of run.sh, but it should work fine for most use cases.
The script looks like the following:
#!/bin/bash
#Simple script to fully run IGRINS PLP on any UTDATE
echo "Enter UTDATE:" #Prompt user to enter date
read UTDATE #Get date from user
#Clean up previous files, if any
rm -r outdata/$UTDATE
rm -r calib/secondary/$UTDATE
rm -r calib/primary/$UTDATE
#Run PLP
python igr_pipe.py flat $UTDATE #Process flat
python ./igr_pipe.py register-sky $UTDATE #First pass at wavelength solution using sky frame (incase THAR lamps don't exist)
python ./igr_pipe.py flexure-setup $UTDATE #Setup files needed for flexure correction
#python ./igr_pipe.py thar $UTDATE #First pass at wavelength solution using ThAr lamps, looks like current PLP version doesn't even
use THAR lamp frames
python ./igr_pipe.py wvlsol-sky $UTDATE #Second pass at wavelength solution using OH sky emission
python ./igr_pipe.py a0v-ab $UTDATE --correct-flexure --mask-cosmics #Reduce A0V standard star data
python ./igr_pipe.py a0v-onoff $UTDATE --correct-flexure --mask-cosmics
#python ./igr_pipe.py tell-wvsol $UTDATE #Use telluric absorption in A0V std star to fine tune wavelength solution
python ./igr_pipe.py stellar-ab $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics #Reduce stellar sources nod on slit
python ./igr_pipe.py stellar-onoff $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics #Reduce stellar sources nod off
slit
python ./igr_pipe.py extended-ab $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics #Reduce extended sources nod on slit
python ./igr_pipe.py extended-onoff $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics #Reduce extended sources not off
slit
python ./igr_pipe.py divide-a0v $UTDATE #Reduce stellar sources nod off slit
#python igr_pipe.py plot-spec $UTDATE --html-output #Make and publish HTML preview
#python igr_pipe.py publish-html $UTDATE #Ditto
echo "Done running"
echo $UTDATE
We will explain each step later.
If this is the first time you have run "run.sh," you will need to give it executable permissions. You only need to do this once.
> chmod +x run.sh
To run the script to reduce a night of IGRINS data, you simply execute it like so:
> ./run.sh
It will ask you for the date of the night you want to reduce. This date will be in the format YYYYMMDD and correspond to both the date for the raw data in the "indata" directory and the recipe file.
> ./run.sh
Enter UTDATE:
2060123
The script will then reduce the entire night automatically and you just need to wait for it to finish. This can take some time, depending on the number of observations taken in that night and how fast your computer is. You will see a lot of output from the PLP telling you what it is doing. When it is done, you will see "DONE RUNNING" with the date.
DONE RUNNING
2016013
>
Here we will explain each step in running the pipeline, using the "run.sh" script as our guide.
The "run.sh" script will start by trying to remove any previous data reduction products for the night you are running with the following:
#Clean up previous files, if any
rm -r outdata/$UTDATE
rm -r calib/secondary/$UTDATE
rm -r calib/primary/$UTDATE
This step is not strictly necessary, as the the PLP will simply overwrite existing files, but exists to clean things up in the "outdata" and "calib" directories before rerunning a night.
The flats are required calibrations for a night. Processing flats is the very first step you need to run in the pipeline to start reducing a night. They are processed in run.sh using:
> python igr_pipe.py flat $UTDATE #Process flat
Registering the sky is the first iteration in the wavelength calibration process and is a required step after the processing the flats. It uses the OH lines in the SKY frame to make a first guess at the wavelength solution:
> python ./igr_pipe.py register-sky $UTDATE #First pass at wavelength solution using sky frame (incase THAR lamps don't exist)
If you are going to use the flexure correction, which is called for the various recipes with the command line argument "--correct-flexure", you need to run the setup script to make sure the necessary files are generated. You only need to run this step if you are going to use the flexure correction. This is done as follows:
> python ./igr_pipe.py flexure-setup $UTDATE #Setup files needed for flexure correction
This runs the second iteration on the wavelength solution using the OH sky emission lines in the SKY frame, using the solution generated by Register as a first guess. This is a required step.
> python ./igr_pipe.py wvlsol-sky $UTDATE #Second pass at wavelength solution using OH sky emission
This step reduces the A0V standard star observations used for telluric correction and flux calibration. Generally each science target will have an associated standard star at a similar airmass. These are normally observed nodded on the slit but can be observed nodded off the slit. The PLP will treat all these as stellar sources and use optimal extraction for the 1D spectra.
> python ./igr_pipe.py a0v-ab $UTDATE --correct-flexure --mask-cosmics #Reduce A0V standard star data
> python ./igr_pipe.py a0v-onoff $UTDATE --correct-flexure --mask-cosmics
These recipes will reduce the various science targets. These include stellar (point) and extended (nebular) sources, both nodded on slit (ab) and off slit (onoff).
python ./igr_pipe.py stellar-ab $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics #Reduce stellar sources nod on slit
python ./igr_pipe.py stellar-onoff $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics #Reduce stellar sources nod off
slit
python ./igr_pipe.py extended-ab $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics #Reduce extended sources nod on slit
python ./igr_pipe.py extended-onoff $UTDATE --correct-flexure --height-2dspec=100 --mask-cosmics #Reduce extended sources not off
slit
The recipes for processing standards and science targets, specifically a0v-ab, a0v-onoff, stellar-ab, stellar-onoff, extended-ab, and extended-onoff, have several options that can be set.
Must be set if you want to turn on the instrumental flexure correction. This will attempt to shift the individual exposures in pixel space in the x-direction to try to keep the OH sky emission lines aligned using the first exposure in the SKY as a reference frame. We highly recommended you use the flexure correction as it normally reduces "p-cygni" type residuals from sky emission and telluric absorption lines. You need to run the "flexure-setup"
Turns on cosmic ray correction for each individual exposure using astroscrappy. The parameters used have been specifically tuned for the IGRINS H and K bands to avoid masking bright sky emission lines or the target spectrum itself. It might not catch all cosmic rays, but it will catch many of the bright ones. The PLP will attempt to "fill" masked cosmics with data from different exposures of the same nod on a target. We generally recommend you leave the cosmic ray masking on. If you require more thorough cosmic ray correction, we recommend you run your own cosmic ray correction on the raw data before reducing the data with the IGRINS PLP.
Sets the height the PLP will interpolate the reduced rectified 2D spectra to in pixels for the spec2d.fits and var2d.fits files found in the "outdata" directory. This is useful when dealing with extended sources and we recommend you use this option. 100 pixels works well, but other values can be used.
The order of pipeline processing is
- flat
- register-sky (use sky frame to do bootstrapping and an initial wavelength solution)
- wvlsol-sky (Sky lines for full wavelength solution)
- a0v-ab (extract spectra for A0V telluric standard stars)
- stellar-ab, extended-ab, extended-onoff
The processed data will be saved under "calib/primary/20140525" and "outdata/20140525". "calib/primary" will contain files required by other recipes. Under the directory "outdata/20140525/qa", several figures should have been generated. It is supposed to help you with the quality assurance, although it is not as complete and descriptive as it meant to be.
For example, to manually process flat for the night of 20140525,
> python ./igr_pipe.py flat 20140525 -s 64 -b "H" This will process "H" band data of a group whose first ObsID is 64 using recipe "FLAT".
There may be multiple observation groups with recipe name of "FLAT", and the default behavior is to process all the groups and both "H" and "k" band. For this, you can simply do
> python ./igr_pipe.py flat 20140525Next is to do "register"
> python ./igr_pipe.py register-sky 20140525 -b "H"As before, this will only process "H" band data. Remove '-b "H"' to process both "H" and "K" bands data.
Next is Sky.
> python ./igr_pipe.py wvlsol-sky 20140525 -b "H"Now, we have all the calibration files ready to extract spectra.
> python ./igr_pipe.py a0v-ab 20140525 -b "H" -s 16This will extract H band spectra of A0V star from a group whose first ObsID is 16 (24 Oph). To extract H- and K- spectra of all A0V stars,
> python ./igr_pipe.py a0v-ab 20140525It is time to extract spectra of target objects.
> python ./igr_pipe.py stellar-ab 20140525 -b "H" -s 11This will extract H-band spectra of a group with starting obsID of 11 (V2247Oph). Again, to extract all target spectra (with recipe name STELLAR_AB).
> python ./igr_pipe.py stellar-ab 20140525Here is some more details on the extraction with available options.
The extracted spectra will be saved under "outdata/20140525" with file name ending with "spec.fits". The file should be compatible with tools like IRAF splot. To quick-look view of the spectrum,
> python ./igr_pipe.py plot-spec 20140525 -b "H" -s 11 -i -mThis will give two matplotlib figures. The first figure shows extracted spectra and its S/N. The second figure (lower panel) is target spectra divided by A0V spectra for quick telluric correction. The upper panel is the spectra of (flattened) A0V spectra for comparison. Note that this is for quick-look only. Without "-i" option, a static figure is saved under the "qa" directory. Supplying "--html-output" option will produce files related to html output under "html/20140525". With "-m" option, a model spectra of Vega is multiplied to the divided spectra.
Similar to "stellar-ab", you can process "extended-ab" and "extended-onoff". The extraction of 1d spectra from extended source is not optimal and needs to be improved. "extended-ab" is for compact extended source which was observed in ABBA mode. "extended-onoff" is for extended source with large extent where a separate off position was observed.
> python ./igr_pipe.py extended-ab 20140525> python ./igr_pipe.py extended-onoff 20140525Here ia a small script that go through all the necessary recipes.
UTDATE=20140525
for RECIPE in flat register-sky wvlosl-sky a0v-ab stellar-ab extended-ab extended-onoff
do
python igr_pipe.py $RECIPE $UTDATE
rc=$?
if [ $rc != "0" ]; then
exit $rc
fi
doneTo produce interactive html page,
UTDATE=20140525
python igr_pipe.py plot-spec $UTDATE --html-output
python igr_pipe.py publish-html $UTDATEIn this case, the last recipe "publish-html" produces the extracted spectra under "html/20140525" as html pages.
To check the spectra, you may need to run a web server under the "html" directory. The easiest way is to use python's SimpleHTTPServer module.
cd html
python -m SimpleHTTPServerThis will run a very simple web server listening to port 8000. Now in your browser, visit "http://localhost:8000/20140525/".
You will see a list of targets observed in 20140525. The links on the right side will give you extracted spectra.
If you have a firefox, you don't need to run a webserver.
> firefox html/20140525/index.htmlYou may be able to do this with other browser. It depends on the security policy of your browser (with chrome, running this way would not give you correctly rendered page because of the default security policy of chrome.)
The current pipeline requires FLAT, THAR, SKY calibrations for each night. Often, some of these calibration files are missing. The easiest workaround is to make a symlink to existing calibration directory. For example, no calibration files are available for 20140712. You may use calibrations from 20140710. What you have to do link "calib/primary/20140710" to "calib/primary/20140712".
> cd calib/primary
> ln -s 20140710 20140712You can now be able to extract spectra of 20140712 using recipes like "A0V_AB", "STELLAR_AB", etc.