This document describes an early prototype of the LSST Alert Generation system. As of mid-2019, much of the material presented here is obsolete. Please refer to the Science Pipelines documentation for current information.
Ultimately, LSST Alert Production (AP) will need to perform all steps of the Level 1 pipeline automatically on each new raw image from the telescope. Details are in the LSST Data Management Science Pipelines Design (LDM-151 :cite:`LDM-151`) and the LSST Data Products Definition Document (DPDD :cite:`LSE-163`). Briefly summarized below and in :ref:`Figure 1<fig-nightly-proc>`, the first portion of these steps are:
- Instrument Signature Removal (ISR), which includes crosstalk and linearity corrections, bad pixel masking, bias subtraction, and flat field division,
- Point Source Function (PSF) and background characterization, which includes source detection and measurement,
- Photometric and World Coordinate System (WCS) calibration, and
- Image Differencing with an appropriate template (which may need constructing) to create Difference Image Analysis (DIA) source catalogs.
The first portion of the Nightly Processing Pipeline overview, from Figure 1 of :cite:`LDM-151`. This encompasses all of "Single Frame Processing" and part of "Alert Detection."
A number of packages in the LSST Software Stack have been developed to perform portions of one or more of these four tasks, but they are not yet linked together in an automated way. This Prototype AP Pipeline is the first step toward the goal of an automatic Level 1 Pipeline that can process a set of raw images through each of these steps and verify that the output is what we expect.
To achieve this, there are two main python modules under development:
ap_pipe and ap_verify.
The former, described here, is responsible for running the Prototype Pipeline. The latter
ap_pipe to verify the output.
We use images from the High Cadence Transient Survey (HiTS) :cite:`for16`, a three-year survey from 2013--2015 which used the Dark Energy Camera (DECam) on the 4 m Blanco telescope at Cerro Tololo Interamerican Observatory (CTIO). The primary science goal of this survey is to detect and follow up optical transients that have characteristic timescales from hours to days, with a particular interest in catching early supernovae. The distribution of the HiTS survey fields in the sky is shown in :ref:`Figure 2<fig-hits>`.
Spatial distribution of the fields observed in the 2013--2015 HiTS campaign (Figure 4 from :cite:`for16`). The fields selected for this dataset are three of the pink ones, which indicate they were visited in both 2014 and 2015. Specifically, they are the pair of partially-overlapping pink fields near the bottom center of the 2014 and 2015 field region and the upper-right-most pink field.
We select HiTS fields
are repeat observations of the same region of sky as
fields, respectively. Each field has 34--36 individual visits over about a month, and each DECam image
consists of 62 CCDs (however, two of these are non-functional).
Most of the visits are in the g-band, some are in the r-band, and a few are in i.
All of these data are in the git-lfs repo ap_verify_hits2015
in raw format, along with the corresponding DECam Master Calibration files and camera defect files.
The data also exist on the
lsst-dev server at
/datasets/decam/_internal/raw/hits with the
calibration files at
/dataset/decam/_internal/calib/cpHits and the camera defects at
/dataset/decam/_internal/calib/bpmDes. The raw images and calibration files were originally obtained
from the NOAO Science Archive by searching for PI Förster.
This tutorial walks a user through using
ap_pipe to run four main processing steps
with the LSST Stack on a portion of the :ref:`Dataset <Dataset>` described above:
ingestImagesDecam.py, from the
obs_decampackage but drawing heavily on
ingestCalibs.py, from the
processCcd.py, from the
imageDifference.py, also from the
The prerequisites for running
- The LSST Science Pipelines
stack with the
lsst_appspackages (a subset of
- A clone of the
You may set all of this up as follows:
.. prompt:: bash setup lsst_apps # If using the lsst-dev server, lsst_apps may be setup using a centrally # available weekly build, but the following packages are not included git clone https://github.com/lsst/obs_decam.git git clone https://github.com/lsst-dm/ap_pipe.git setup -k -r obs_decam setup -k -r ap_pipe git clone https://github.com/lsst/ap_verify_hits2015.git
Once you are ready, run
ap_pipe from the command line. You must point to the dataset
-d flag, a desired output location on disk with the
-o flag, and provide
a valid visit and ccdnum dataId string with the
.. prompt:: bash python ap_pipe/bin.src/ap_pipe.py -d ap_verify_hits2015/ -o output_dir -i "visit=410985 ccdnum=25"
At present (DM-11390), the template used for
difference imaging is hard-wired to visit 410929 and ccdnum 25.
This is a single CCD only of one of the
Blind15A_40 visits. If you would like to use
a different template, you must manually set this in the source code
ap_pipe/python/lsst/ap/pipe/ap_pipe.py, in the function
This functionality will be improved when we switch from using a visit as a template to using coadds
by default (see DM-11422).
This will run all four steps of the pipeline in order. Each step will create a new subdirectory
output_dir: one for ingested images, one for ingested calibration products, one for
processed images, and one for difference imaging.
The difference image and DIA Source catalog for a test image are available
lsst-dev server at
A screenshot of the calibrated exposure (calexp) and difference image is shown
in :ref:`Figure 3<fig-diffim>`.
Calibrated science exposure (top) and difference image (bottom) for CCD 25 of HiTS field
This example uses visit 410929 as the template image and visit 410985 as the science image.
Future work will use coadded 2014 visits of the three HiTS fields described in Dataset
as templates and the corresponding 2015 visits as science.
This pipeline is a core component of the
AP Minimum Viable System
(note this link is private to LSST team members only).
The goal is to run and verify each component of LSST image processing as we build a
ap_pipe pipeline will also be available to users who wish to process any DECam dataset independently.
.. bibliography:: local.bib :style: lsst_aa