-
Notifications
You must be signed in to change notification settings - Fork 0
Introduction to the Data Reduction Manager
The Data Reduction Manager is a widget-based, graphical user interface to the Interactive Data Language (IDL) command line , specifically for the reduction of astronomical images. In its most basic form, DRM may also be used as a tool for FITS file management and image manipulation.
Start DRM via 'idl drm_new' on the command line, or typing '@drm_new' from within IDL.
DRM allows the user to go from raw astronomical images to fully calibrated cylindrical maps of the radiance from the planet. Those radiance maps can then be used to derive atmospheric properties such as temperatures, clouds and gaseous composition of the planets. The basic steps are shown in the diagram below, but may vary from instrument to instrument.
Steps include:
- Taking each 'frame', subtracting the background sky emission.
- Correcting for both the array response, bad pixels and electronic striping.
- Coadding images to improve signal to noise.
- Deblurring the image - removing the diffraction and seeing effects.
- Assigning latitudes and longitudes (the geometry) to each point on the planet.
- Creating cylindrical lat/lon maps of the radiance.
- Calibrating the radiance (by scaling to be the same as spacecraft data).
- Stacking multiple filtered images to define a 'pseudo-spectrum' for retrievals of temperature and composition.
DRM allows the user to go from raw astronomical images to fully calibrated cylindrical maps of the radiance from the planet. Those radiance maps can then be used to derive atmospheric properties such as temperatures, clouds and gaseous composition of the planets. Data reduction is the decoding of the raw data, the reduction in the original number of data elements and the formation of a calibrated image.
FITS Images: All images are stored in the Flexible Image Transport System (FITS) is a graphics format designed for the exchange of astronomical data. Users must utilise other resources to read and display the stored data. The common format facilitates the exchange of data between telescopes, professional astronomers, etc. A generic FITS file is composed of a header, which describes both the header and the format of the contents, and the data itself.
Working files (i.e. containing the target images): wmYYMMDD_####.*.gz These are the working files, #### indicated the file numbers, * indicates the chop/nod position (a,b,c,d,e...).
Bad pixel mask: badpixelmask.gz: a map of the detector array with the corrupt pixels zeroed-out. Bad pixels are caused by random imperfections in the manufacturing process, damage to the detector, age or equally cosmic rays and saturation from a bright star. Although there are a number of ways these masks may be determined, bad pixels are most readily identified by dividing a short exposure of an evenly illuminated source by a long exposure. The bad pixel mask should not change much over time, so you may use one from an adjacent date.
Flat-field images: flat_17.2.gz: images of the telescope dome, useful for determinging the variation in pixel sensitivity at each pixel over the whole image. Although the flat fields are specific to a particular date, we do not have time to take them each time - search back through the directories and use the most recent.
In file names, you may see the following single letters:
s means saturn,
j means jupiter.
followed by 'c' for coadd;
'f' for filtered (sometimes more than once);
'p' for PIXON inversion.
'cmap' denotes a cylindrically mapped file.
'mu' is a file containing the emission angles for a particular image.
'cal_' prefix denotes an absolutely calibrated image, e.g. cal_scf######.cmap.gz / .gz /.mu.gz
'wd' means weighted-deconvolved (the equivalent of the non-negative first-stage of a PIXON inversion).
When we coadd lots of files, the new filename includes the number of start file and the end file in the sequence, e.g. scfxxx-yyy.gz where xxx was the first file, yyy was the second file.
Start IDL by typing 'idl' on the command line. Start DRM by typing @drm_new at the idl prompt. Select a color table to see nicer images: Options -> Load color table -> Select a color table -> Done. Use 'Get Files' and navigate the JPL directory structure to an example of a FITS file. 'Select' the FITS file you want to view, it'll now appear in the left column of DRM's main window. Double-click that filename -> it'll now appear in its own browser window. The 'Tools' and 'Programs' menues at the top of the page will allow you to perform the numerous reduction and calibration tasks required for these images! To save any of your results, use Save -> All Selected gives you the option of adding a new prefix to explain the steps taken for each image.
To facilitate convenient exchange of data between telescopes, professional astronomers typically use the Flexible image transport system, FITS, to store their data. A generic FITS file is composed of a header that describes the format of the contents and useful metadata, and the data itself. Preprocessing reads the input FITS files, groups them according to target and wavelength, re-calculates the LCM and airmass and stores these values within the header. For example, this would include information on the distance of the target and the plate scale of the imager as needed by the ‘Geometry’ program to fit limbs to the planetary disc. This is done by running the ‘Preprocessing’ program on the image fits files.
- Get the files, and select them all into the right-hand window.
- Go to Programs > Preprocessing > Preprocess all selected.... and wait for it to complete.
Note that to use the newly processed files, you'll need to "unget" the ones you already have, then reselect the files from the newly formed wavelength/target subdirectories.
Whether due to light scattering, thermal interference or otherwise, observations typically include some low luminosity component which must be removed. Named after the beam-switch observing pattern it takes advantage of, ABAB subtraction removes the background sky emission on which the target's own emission is superimposed (usually as a very small component).
By rapidly alternating the telescope's secondary mirror to offset the image between two different points on the detector (chopping), the local sky background may be subtracted from the sky + target measurement, leaving only those photons from the target (the A and B chopped pair). The act of chopping alone is however only sufficient in removing the bulk of the thermal emission. As the optical path through the telescope is slightly different for the two chop positions, linear spatial variations in the background sky emission persist and need to be corrected for. This is accomplished by offsetting the telescope to another point on the sky (nodding) and subtracting the new chopped AB pair from the original chopped AB pair. An added benefit of such sky subtraction, is that the internal electronics deal with the beams separately, thereby minimizing the systematic bias introduced by the instrument during observations.
Chopping refers to a moving secondary mirror while the primary remains fixed on the object; we use nodding to indicate a fixed secondary where the pointing of the primary mirror alternates between sky and an object field.
Having removed the bulk of the sky background using ABAB subtraction, we use a second sky subtraction step to minimise any linear variations across the array, If the sky emission were spatially uniform, then chopping alone would be sufficient to recover the flux from the target. Sky subtraction is more important at the longer wavelengths. These sky only images capture the thermal emission of the sky background.
Dividing by the flat field minimises the effect of non-uniformities in the detector array response.
Despiking means we take a look at a region surrounding each pixel, and replace that pixel with a mean value if the pixel value exceeds N sigma of the mean. This smooths the image and removes spikes, making the planet much easier to see.
Depatterning attempts to account for the amplification noise which results from the way the detector array is read out by the electronics. For MIRSI (and many other detector arrays), the read noise pattern is not randomly distributed across the array but appears as regular, 20-pixel wide columns, across the entire width of the image.
Co-adding the images increases the signal-to-noise ratio by the square root of the number of integrations -- in principle, by as much as four times for a four-step, chop-nod (two by two) sequence. To co-add, the images must first be suitably aligned. This is usually accomplished by either via the astro-correlate method or the geometry method. The geometry method is the more accurate alignment method of the two, but also the more labour intensive.
Once all images have been coadded, a fourier filter can be applied to smooth the image based on its spatial frequency components. The image is fourier transformed and spatial frequencies above a specified limit are discarded. This has the effect of removing high frequency noise from the image.
To assign latitudes, longitudes and emission angles to the data, we must fit the planetary limb and store the resulting geometry values in the header. This should be done irrespective of whether geometry was assigned during the coaddition stages.
Derive the radiometric intensity and store it within the header. NOTE: It is easy to confuse photometry with radiometry (as the DRM authors have below), but there is an important difference. Whereas radiometry includes the entire optical spectrum (encompassing the UV, visible and IR), photometry is limited to the visible spectrum as defined by the spectral response of the eye.
Additional Note: The astronomical research community often uses these two terms interchangeably. So if an modern astronomer says 'photometry', they probably mean 'radiometry' and will not be using any corrections for the human eye.
Having successfully reduced the data, project the positive image into a cylindrical map ("cylmap") allowing for ease of visualisation and ready inter-comparison with other products.
The final step in the processing of ground-based astronomical images is the assignment of realistic radiance values, as if the data were measured by an ideal detector above the atmosphere. Incorporating telluric correction, such "absolute" calibration is accomplished by dividing the image by that of a nearby source (one of the so-called standard stars whose flux (F) has been determined absolutely) and multiplying by the standard's known radiance (L)
Instead of employing stars however, the absolute calibration employed here is accomplished by radiometrically scaling the data to values based on the zonally averaged radiances measured by Cassini/CIRS and/or Voyager/IRIS. For the respective datasets (i) and each spectral band (j), the in-band radiance (L_ij) may be calculated:
where L_ij is the zonally averaged radiances and f_j is the relative flux as a function of wavenumber, v. In this way, the measurements are converted into absolute radiance units [ergs/(s sr cm2 cm-1)], and may be readily used for inter-comparison.
NOTE: Telluric correction accounts for the variable absorption due to Earth's atmosphere. Although the atmospheric water content can change significantly with time, the frequency dependence is assumed to be conserved. This remains valid despite the oscillating attenuation, as constant scale factors cancel out in the normalisation accomplished to obtain the relative fluxes.