There may be cases where you would be interested in accessing the DRAGONS Application Program Interface (API) directly instead of using the command line wrappers to reduce your data. Here we show you how to do the same reduction we did in the previous chapter but using the API.
If you have not already, download and unpack the tutorial's data package. Refer to datasetup
for the links and simple instructions.
The dataset specific to this example is described in:
about_data_set
.
Here is a copy of the table for quick reference.
Science | S20131121S0075-083 |
Y-band, 120 s |
|
S20131121S0369-375
---------------------+ | S20131120S0115-120 | S20131121S0010 | S20131122S0012 | S20131122S0438-439 |
|
|
S20131129S0320-323
---------------------+ | S20131126S1111-116 |
|
We first import the necessary modules and classes:
import glob
from gempy.adlibrary import dataselect
from recipe_system import cal_service
from recipe_system.reduction.coreReduce import Reduce
glob
is a Python built-in package. It will be used to return a list
with the input file names.
~gempy.adlibrary.dataselect
will be used to create file lists for the darks, the flats and the science observations. The ~recipe_system.cal_service
package is our interface with the local calibration database. Finally, the ~recipe_system.reduction.coreReduce.Reduce
class is used to set up and run the data reduction.
We recommend using the DRAGONS logger. (See also double_messaging
.)
from gempy.utils import logutils
logutils.config(file_name='f2_data_reduction.log')
Before we continue, let's be sure we have properly setup our calibration database and the calibration association service.
First, check that you have already a rsys.cfg
file inside the ~/.geminidr/
. It should contain:
[calibs]
standalone = True
database_dir = ${path_to_my_data}/f2img_tutorial/playground
This tells the system where to put the calibration database. This database will keep track of the processed calibrations as we add them to it.
Note
The tilde (~
) in the path above refers to your home directory. Also, mind the dot in .geminidr
.
The calibration database is initialized and the calibration service is configured as follow:
caldb = cal_service.CalibrationService()
caldb.config()
caldb.init()
cal_service.set_calservice()
The calibration service is now ready to use. If you need more details, check the section on using the API in the .
Next step is to create lists of files that will be used as input to each of the data reduction steps. Let us start by creating a list
of all the FITS files in the directory ../playdata/
.
all_files = glob.glob('../playdata/*.fits')
all_files.sort()
The ~list.sort
method simply re-organize the list with the file names and is an optional step. Before you carry on, you might want to do print(all_files)
to check if they were properly read.
Now we can use the all_files
list
as an input to ~gempy.adlibrary.dataselect.select_data
. The dataselect.select_data()
function signature is:
select_data(inputs, tags=[], xtags=[], expression='True')
We select the files that will be used to create a master dark for the science observations, those with an exposure time of 120 seconds.
dark_files_120s = dataselect.select_data(
all_files,
['F2', 'DARK', 'RAW'],
[],
dataselect.expr_parser('exposure_time==120')
)
Above we are requesting data with tags F2
, DARK
, and RAW
, though since we only have F2 raw data in the directory, DARK
would be sufficient in this case. We are not excluding any tags, as represented by the empty list []
. The expression setting the exposure time criterion needs to be processed through the dataselect
expression parser, ~gempy.adlibrary.dataselect.expr_parser
.
We repeat the same syntax for the 2-second darks:
dark_files_2s = dataselect.select_data(
all_files,
['F2', 'DARK', 'RAW'],
[],
dataselect.expr_parser('exposure_time==2')
)
Now you must create a list of FLAT images for each filter. The expression specifying the filter name is needed only if you have data from multiple filters. It is not really needed in this case.
list_of_flats_Y = dataselect.select_data(
all_files,
['FLAT'],
[],
dataselect.expr_parser('filter_name=="Y"')
)
Finally, the science data can be selected using:
list_of_science_images = dataselect.select_data(
all_files,
['F2'],
[],
dataselect.expr_parser('(observation_class=="science" and filter_name=="Y")')
)
The filter name is not really needed in this case since there are only Y-band frames, but it shows how you could have two selection criteria in the expression.
We first create the master dark for the science target, then add it to the calibration database. The name of the output master dark is N20160102S0423_dark.fits
. The output is written to disk and its name is stored in the Reduce instance. The calibration service expects the name of a file on disk.
reduce_darks = Reduce()
reduce_darks.files.extend(dark_files_120s)
reduce_darks.runr()
caldb.add_cal(reduce_darks.output_filenames[0])
The ~recipe_system.reduction.coreReduce.Reduce
class is our reduction "controller". This is where we collect all the information necessary for the reduction. In this case, the only information necessary is the list of input files which we add to the files
attribute. The ~recipe_system.reduction.coreReduce.Reduce.runr
method is where the recipe search is triggered and where it is executed.
Note
The file name of the output processed dark is the file name of the first file in the list with _dark appended as a suffix. This is the general naming scheme used by the Recipe System
.
By default, for F2 imaging data, an illumination mask will be added to the data quality plane to identify the pixels beyond the circular aperture as "non-illuminated". The package does not have a default bad pixel mask for F2 but the user can easily create a fresh bad pixel mask from the flats and recent short darks.
The Bad Pixel Mask is created using as follow:
reduce_bpm = Reduce()
reduce_bpm.files.extend(list_of_flats_Y)
reduce_bpm.files.extend(dark_files_2s)
reduce_bpm.recipename = 'makeProcessedBPM'
reduce_bpm.runr()
bpm_filename = reduce_bpm.output_filenames[0]
The flats must be passed first to the input list to ensure that the recipe library associated with F2 flats is selected. We are setting the recipe name to makeProcessedBPM
to select that recipe from the recipe library instead of the using the default (which would create a master flat).
The BPM produced is named S20131129S0320_bpm.fits
.
The local calibration manager does not yet support BPMs so we cannot add it to the database. It is a future feature. Until then we have to pass it manually to the Reduce
instance to use it, as we will show below.
A F2 master flat is created from a series of lamp-on and lamp-off exposures. Each flavor is stacked, then the lamp-off stack is subtracted from the lamp-on stack and the result normalized.
We create the master flat field and add it to the calibration manager as follows:
reduce_flats = Reduce()
reduce_flats.files.extend(list_of_flats_Y)
reduce_flats.uparms = [('addDQ:user_bpm', bpm_filename)]
reduce_flats.runr()
caldb.add_cal(reduce_flats.output_filenames[0])
Note how we pass in the BPM we created in the previous step. The addDQ
primitive, one of the primitives in the recipe, has an input parameter named user_bpm
. We assign our BPM to that input parameter. The value of uparms
needs to be a list
of Tuples
.
Once runr()
is finished, we add the master flat to the calibration manager (line 59).
The science observation uses a dither-on-target pattern. The sky frames will be derived automatically for each science frame from the dithered frames.
The master dark and the master flat will be retrieved automatically from the local calibration database. Again, the user BPM needs to be specified as the user_bpm
argument to addDQ
.
We use similar commands as before to initiate a new reduction to reduce the science data:
reduce_target = Reduce()
reduce_target.files.extend(list_of_science_images)
reduce_target.uparms = [('addDQ:user_bpm', bpm_filename)]
reduce_target.runr()
The output stack units are in electrons (header keyword BUNIT=electrons). The output stack is stored in a multi-extension FITS (MEF) file. The science signal is in the "SCI" extension, the variance is in the "VAR" extension, and the data quality plane (mask) is in the "DQ" extension.