Routines for subtracting sky lines from Hectospec spectra, particularly in cases of strong background emission.
Hectosky is an IDL-based program for sky subtraction of spectra taken with the multi-fiber spectrograph Hectospec. Hectosky optimizes the subtraction of spatially-variable nebular emission lines while minimizing the noise introduced by subtracting the sky emission lines.
Hectosky is currently optimized to be run after the E-SPECROAD data reduction pipeline, which was written by Juan Cabanela at Minnesota State University and is available at:
performs its own basic sky subtraction and places the results in
PointingName" is the name of a Hectospec
pointing, found in the raw file names as
Do not use the sky subtraction from E-SPECROAD if you are using Hectosky. Instead, Hectosky will use the multispec files
along with the 1D spectra in the directory
1d.PointingName/ and the
The code for Hectosky and its associated programs is heavily commented, so if you run into something not described in this file, looking at the code may be helpful.
Sky Subtraction Outline
Screens sky fibers for any usuable skies (e.g. sky fibers that landed on a star).
Combines all usable skies into a master sky.
Fits nebular emission lines like H-alpha in the master sky and subtracts the fits to create a master-minus-nebular template.
Subtracts the master-minus-nebular spectrum from the individual sky spectrum closest on the sky to a science object, creating a nebular-lines-only sky spectrum.
Fits the nebular lines in the nebular-lines-only spectrum.
Adds the lines fit from the nebular-lines-only to the master-minus-nebular to create a synthetic sky. The synthetic sky thus consists of the continuum and OH, O2 and other sky emission lines from the master sky, plus the nebular lines corresponding to their strength in the nearest individual sky spectrum.
Subtracts the synthetic sky from the science object spectrum.
The IDL Astronomy User's Library (http://idlastro.gsfc.nasa.gov/)
MPFIT, an IDL-based curve fitting program created by Craig Markwardt (http://cow.physics.wisc.edu/~craigm/idl/fitting.html)
IRAF (available at http://iraf.noao.edu/)
WCSToolspackage for IRAF (available via ftp as described by http://tdc-www.harvard.edu/software/wcstools/iraf.install.html)
hectospecpackage for IRAF (available at http://tdc-www.harvard.edu/iraf/hectospec/)
The Hectosky package has IDL and bash components.
The IDL components of Hectosky are the programs:
hectosky.pro(and its variations)
These should be placed in a folder in your IDL path. I have a folder
in my home directory called
idl and a line in my
.bashrc that adds
this directory to my IDL_PATH:
The bash components are the programs:
These should be placed in a folder in your bash PATH. I have a folder
in my home directory called
bin and a line in my
.bashrc that adds
this directory to my PATH:
Setting up these three scripts is the most complicated part of getting Hectosky to run and requires a basic understanding of your IRAF installation. There exists a way to query the required parameters within the bash script, so that they don't all have to be set manually, but this version of Hectosky does not have that capability.
calloned in your favorite text editor. You will need to edit
two lines of the header as follows:
Replace the path above with the path to the
ecl.e file of your IRAF
set arch = .linux64
Replace the '.linux64' with your system's architecture. To determine what
system architecture your IRAF installation is using, enter the IRAF
environment and execute:
cl> show arch
To test if the parameters are correctly set on your system, go to a directory with a FITS spectrum and execute (replacing "spectrum" with the name of your file):
$ xgterm -e calloned splot spectrum.fits
splot command works and you see your spectrum plotted, then you're good
to go. If not, open an
xgterm terminal and execute:
$ calloned splot spectrum.fits
This will allow you to see any error messages that result. When I
first set up these programs on my computer, there was something weird
about the IRAF distribution (v2.14.1) that was causing problems: IRAF was
looking for the
x_onedspec.e file in the wrong directory. I copied
x_onedspec.e to the folder it was searching, and the problem was
solved. (I haven't had this problem with IRAF v2.16.)
Once you have
calloned working, edit
callimutil in the same way. The Hectosky package is now ready to run.
Hectosky should be run from the directory with your reduced Hectospec data.
To start, enter IDL and execute
PointingName is the name you gave to the Hectospec pointing.)
Optionally, you can execute
to tell the program to proceed without stopping to review the sky subtraction
of individual objects. It will not be entirely automatic, as there is an
interactive portion that cannot be skipped, but it will be faster. I recommend
not using the
quick option on the first run through a data set; take advantage of
Hectosky's display features to get a feel for the sky subtraction quality.
Where to save output
By default, Hectosky saves sky-subtracted spectra to
If a directory with that name already exists in your working directory, Hectosky will ask
Sky-subtracted spectra already exist in PointingName.skysub/ Replace it? (y/n)
y answer will delete the contents of the existing folder in preparation
for the new sky-subtracted files. A
n answer will prompt you for a new
output folder name:
Enter new folder name for sky-subtracted results:
IMPORTANT: The folder name MUST end with a forward slash (
or the files won't save to the right place.
Checking for existing sky data
Hectosky begins by checking for 1D sky offset spectra. E-SPECROAD splits the multispec object file into 1D spectra, but not the skies, so Hectosky will do the splitting if needed. An xgterm window will open and print its progress (this takes maybe 10 seconds). There is no user input during this process.
Before calling GetGoodSky to screen for usable sky spectra,
Hectosky checks if the list
PointingName.good_sky_data.txt already exists in your current directory. This
allows you to skip the time-consuming process of screening the sky
fibers if you've already done it satisfactorily once. If the file
exists, you will be asked
List of good sky spectra detected. Use this? (y/n)
y answer tells Hectosky to skip calling GetGoodSky and use the
information from the existing file instead. A
n answer deletes the
existing file, to be replaced by a new list of workable sky spectra.
Similarly, if there is already a master sky named
in your working directory, Hectosky will ask
Master sky file detected. Use this? (y/n)
Again, if you answer
n, the existing file will be deleted.
GetGoodSky & creating the master sky
Hectosky calls its companion program, GetGoodSky, to check the sky fibers and screen out ones that cannot be used (e.g., because they landed on a star or have a cosmic ray on top of a nebular line). GetGoodSky loops over all the sky fibers (all the fibers from a sky offset pointing plus the dedicated sky fibers from the science pointing) and uses MPFIT to fit the Hα emission line with a Gaussian.
GetGoodSky will halt and ask for user input if it encounters sky spectra with one or more of the following properties:
A continuum level (default between 5150 and 5400 Å) more than 2σ above the median. This often signals a sky contaminated by stellar light.
Central wavelength of the Hα fit is more than 3σ away from the median. This can indicate the presence of a cosmic ray or other oddity in the Hα line.
For spectra taken with the 600 lpm grating, a full width half max (FWHM) of the Hα fit ≥ 3 Å. For spectra taken with the 270 lpm grating, a FWHM more than 1σ away from the median. This usually signals a poorly-fit Hα line and is primarily of interest if you want reasonably accurate Hα measurements in order to make an Hα map. If you don't care about the accuracy of the Hα equivalent width measurements, you can mostly ignore this warning.
A positive equivalent width, i.e., Hα has been fit as an absorption line. Usually these are clear stellar contaminants.
(The means/medians/sigmas are calculated from all the sky spectrum from that pointing.)
GetGoodSky will plot the full flagged spectrum with the Gaussian fit
of Hα overplot in red, print the reasons the spectrum was flagged
to the termainal, and ask the user the following questions. Questions must be
answered with a
y (for yes) or
n (for no), followed by the
If you type in anything else and hit
Enter, the program will repeat
the question and otherwise do nothing.
Zoom in on H-alpha line? (y or n)
y will replot the spectrum and the Gaussian fit between 6200 and
6900 Å. The dashed line shows where the center of the Hα line is
expected to be based on the mean of the fits to all the skies.
n will send you to the next question without replotting.
Keep this spectrum as is? (y or n)
If, in your judgement, there is nothing wrong with the sky spectrum and you do not wish to
refit the Hα line, hit
y. GetGoodSky will consider this sky
spectum usable and continue its loop over the skies. A
will send you to the next question.
Refit in SPLOT? (y or n)
If the sky spectrum is good but the Gaussian fit is poor (and you want
to improve it), or if you want to be able to zoom in further on the
spectrum before making a decision, hit
y. This will open an IRAF
graph window with the sky spectrum plotted. All SPLOT functions work
as usual. Use the 'd' key and subsequent prompts to fit the Hα
line. You can repeat the fit as often as you like; GetGoodSky will
take the last one.
When you're satisfied with your fit in SPLOT (or have decided that the spectrum is unusable), hit 'q' to hit SPLOT and return to the IDL terminal. GetGoodSky will print the parameters of your fit and ask
Did you successfully fit the line? (y or n)
y will mark the sky as usable and keep your fit; a
n will mark
the sky as unusable and move on.
If you chose not to refit with SPLOT, you will be prompted with a final question:
Set EW to zero? (y or n)
This option exists for the few cases I have encountered where, within
the noise, there is no Hα line emission but the sky is still
usuable for sky subtraction. Answering
mark the sky as usuable;
n will mark the sky as unusable.
When GetGoodSky has fit and checked all the sky spectra, it prints the
list of usable skies to a file (
and returns control to Hectosky.
Hectosky then makes a median master sky out of the good sky fibers, scaling
by exposure time where necessary. The master sky is saved as
Fitting nebular lines in the master sky
Next, Hectosky fits the nebular sky emission lines in the master sky,
starting with Hα. The fitting routine sometimes has trouble
getting the Hαfit right the first time, especially for the
lower-resolution 270 gpm spectra. If you answer
n to the question
Use this fit to the H-alpha line? (y/n)
a SPLOT window will open up, where you can manually refit the line, again using the 'd' key and subsequent commands. When you quit out of SPLOT, Hectosky will refit the Hα line using the SPLOT fit as an initial guess, then replot and repeat its question. If you are satisfied with the fit to the Hα line in the master sky at this point, the program will move on to the rest of the nebular lines.
Hectosky will try to fit all of the following lines that fall within the wavelength range of your spectra:
- Hα, Hβ
- [NII] λλ 6547, 6584
- [SII] λλ 6717, 6731
- [ArIII] λ 7135
- He I λλ 5876, 6678, 7065
- [OIII] λλ 4959, 5007
Except for Hα, you do not have the option to refit these lines.
If the fit is poor or the line does not clearly appear in the master
sky spectrum, answering
Include the fit to this line? (y/n)
means that line will not be fit as a nebular line for sky subtraction.
NOTE: Pay careful attention to the dotted vertical line showing the expected central wavelength for the line in question. If the line is very small, sometimes Hectosky will fit to a nearby sky emission line, and the fit will look good, but be wrong.
ALSO NOTE: At the resolution of Hectospec, the emission lines are not always perfectly Gaussian. Do not worry if there is a slight deviation or if the Gaussian fit appears to have a slightly higher peak flux than the line in the master sky. This effect will largely be cancelled out by the second line fitting later in the program.
After going through the nebular line fits, Hectosky plots the master sky spectrum with all including nebular line fits overplotted. The TOP panel shows the full wavelength range, while the BOTTOM panel zooms in on the 6500-6800 Å region to emphasize the Hα, [NII], and [SII] fits. The terminal will say:
When you have finished reviewing the master sky, hit ENTER to continue to sky subtraction of individual spectra.
Reviewing the sky subtraction
Hectosky performs sky subtraction on all object spectra and on all dedicated sky fibers from the science exposures (but not for the spectra from sky offset exposures). The latter are included as a way to check the accuracy of the sky subtraction (i.e., a sky-subtracted sky should be basically a flat line at zero counts).
For each spectrum, the IDL terminal prints
This is: name Hit ENTER to continue to the next object.
where "name" is "sky" for a dedicated sky fiber and the object name/ID for a science source.
All of the other information about the sky subtraction is contained in the large IDL plot window, which is labeled with the aperture number (for those who, like me, use fiber numbers to keep track of spectra from a certain night). The window has four plots, all of which have wavelength on the x-axis and counts on the y-axis.
The plots on the LEFT show the sky spectrum associated with this object ("associated" means either the sky offset for that fiber, or, if that sky was deemed unusable, the nearest sky fiber on the sky). The actual observed sky spectrum is plotted in white, while the "synthetic sky," composed of the master sky + nebular emission lines specifically fit to this sky, is in red. The green line is the residual and should be flat and close to zero except for cosmic rays.
The plots on the RIGHT show the object spectrum, in white before sky subtraction and in teal after.
To evaluate the sky subtraction, first look at the plots on the LEFT. Is the green line (the difference between the real and synthetic skies) flat and close to zero? Check the BOTTOM LEFT panel to see if there were any problems in fitting the Hα and [NII] lines. If you screened the sky fibers carefully, there shouldn't be too many problems, but occasionally sometime weird slips through. A low-level, double-peaked pattern in the center of the lines in the residual is normal; it is caused by the nebular lines not being perfect Gaussians at the resolution of Hectospec.
Next, look at the plots on the RIGHT. Check the TOP RIGHT plot for the overall sky-subtracted spectrum. For faint stars, verify that you can actually see the stellar continuum. For all stars, check for any strange features in the teal (sky-subtracted) spectrum. Emission spikes may remain at 5577, 6300, and 6363 Å: these are [OI] auroral lines and vary too rapidly with time to be accurately removed in fiber spectra.
The BOTTOM RIGHT panel shows a close-up of the object spectrum around the Hα line. Note that the two BOTTOM plots are on the same scale, allowing you to compare the magnitude of the sky Hα with the size of any features in the object spectrum. Look particularly at the [NII] lines. For nebular emission, the [NII] line strengths scale fairly closely with the Hα line strength. For stellar spectra in which you do not expect any [NII] absorption or emission, you can therefore use any residual [NII] in the stellar spectrum to help you understand any residual Hα If there appear to be [NII] absorption lines, the Hα line is oversubtracted: the real line in the object is not as deep as it appears. If there appear to be [NII] emission lines, the Hα line is undersubtracted and you should be cautious about interpreting any apparent Hα emission.
- FITS files of sky-subtracted science spectra, saved to a folder (default
PointingName.skysub/) in the working directory.
- FITS file of the master sky spectrum (
PointingName.mastersky.fits) unless you chose to use a previously-existing master sky.
- A text file with information on the usable sky fibers (
PointingName.good_sky_data.txt). In addition to recording the sky fibers that went into the master sky, this file gives the equivalent width (EW), full width half max (FWHM), and central wavelength of the Gaussian fit to the Hα line in each usable sky spectrum. A description of each column is given in the file header.
There are three alternate versions of Hectosky: Hectosky_n2fix, Hectosky_s2comp, and Hectosky_one.
Hectosky_n2fix fixes the wavelength of the [NII] λ 6547 line during fitting. Occasionally, this line is clearly visible in the master sky, but the program has trouble fitting it. Hectosky_n2fix corrects this problem and should be used in these cases.
Hectosky_s2comp is optimized for working with spectra from regions like the Orion Nebula, which have some strong nebular lines but minimal [NII] emission. The spectra in Hectosky_s2comp are plotted so that the [SII] lines can be compared to Hα. Extra caution should be used when interpreting apparent Hα emission features.
Hectosky_one does sky subtraction of a single fiber from a
Hectospec pointing, allowing you to fiddle with the subtraction for
a particular object. Sometimes, the difference in nebular line strength between the sky
offset and the science spectrum is large enough that the nebular lines
need to be scaled by some factor to improved sky subtraction.
Hectosky_one has extra input variables and is called by the command:
aperture is the fiber number of the object to be sky-subtracted from
scalefactor is the value by which to
scale the nebular line strengths. e.g.,
It is recommended that you use Hectosky_one only (1) on particularly interesting sources, because it can get quite tedious, and (2) when you are comfortable with using the basic version of Hectosky and have a good feel for what "good" sky subtraction looks like.
Copyright 2012, 2015 Megan Kiminki
Released under the terms of the MIT license. See the
LICENSE file for details.
If you find Hectosky to be useful in your research, please cite Kiminki et al. 2015, ApJ, 813, 42.
Hectosky was developed with support from the National Science Foundation through Astronomy and Astrophysics Research Grant AST-0907980.