Add content to overview, getting started, and tutorial docs

Includes note about decompressing and extracting certain files,
note about decam defect ingestion, and consistent section headers.

Also adds ap_pipe executable to ups table file.

doc/ap_pipe/index.rst

@@ -17,7 +17,7 @@ Repository
    https://github.com/lsst-dm/ap_pipe
 
 JIRA component
-   ``ap_pipe``
+   `ap_pipe <https://jira.lsstcorp.org/issues/?jql=project %3D DM AND component %3D ap_pipe>`_
 
 Modules
 =======

doc/lsst.ap.pipe/getting-started.rst

@@ -4,28 +4,100 @@
 Getting started with the AP Pipeline
 ####################################
 
-`lsst.ap.pipe` is not yet included with the LSST Science Pipelines stack.
-You will need to clone and set it up by following the directions
+
+.. _section-ap-pipe-installation:
+
+Installation
+============
+
+`lsst.ap.pipe`, available `here <https://github.com/lsst-dm/ap_pipe>`_,
+is not yet included with the LSST Science Pipelines stack.
+You will need to clone, setup, and build it by following the directions
 `here <https://pipelines.lsst.io/install/package-development.html>`_.
 
 .. note::
 
-   `lsst.ap.association` is a prerequisite for `lsst.ap.pipe`.
-   You must clone ``ap_association`` and set it up first.
+   `lsst.ap.association`, available `here <https://github.com/lsst-dm/ap_association>`_,
+   is a prerequisite for `lsst.ap.pipe` and is also not currently part of the
+   LSST Stack. You must clone ``ap_association`` and set it up first.
+
+
+.. _section-ap-pipe-ingesting-data-files:
+
+Ingesting data files
+====================
+
+LSST-style image processing typically operates on Butler repositories and does not
+directly interface with data files. `lsst.ap.pipe` is no exception.
+The process of turning a set of raw data files and corresponding calibration
+products into a format the Butler understands is called ingestion. Ingestion
+can be somewhat camera-specific, and is outside the scope of the AP Pipeline.
+
+.. TODO: Cut or condense above paragraph and link to ingestion-related docs.
+
+A utility to ingest data before running `lsst.ap.pipe` 
+is available in :ref:`ap-verify-run-ingest`. However, this works
+only on datasets which adhere to the :ref:`ap-verify-datasets` format.
+Alternately, you may use a pre-
+ingested dataset or manually ingest files yourself following the directions
+for a given ``obs_`` package, e.g.,
+step 4 of `the obs_decam README <https://github.com/lsst/obs_decam/blob/master/README.md>`_.
+
+A standard ingestion workflow for DECam looks something like
+
+.. prompt:: bash
+
+   ingestImagesDecam.py input_loc --filetype raw path/to/raw/files
+   ingestCalibs.py input_loc --calib calib_loc path/to/flats/and/biases --validity 999
+   ingestCalibs.py input_loc --calib calib_loc --calibType defect --mode=skip path/to/defects --validity 0
+
+.. note::
+
+   Defect ingestion is a step unique to DECam. It presently requires 
+   ``--mode=skip``, this mode interprets paths as relative to ``calib_loc``,
+   and the validity value is not used (but must be included). This interface
+   may change when `DM-5467 <https://jira.lsstcorp.org/browse/DM-5467>`_ is completed.
+
+
+.. _section-ap-pipe-required-data-products:
+
+Required data products
+======================
+
+For the AP Pipeline to successfully process data, the following is required:
+
+- **Raw science images** and `reference catalogs 
+  <https://community.lsst.org/t/creating-and-using-new-style-reference-catalogs/1523>`_
+  ingested into a main Butler repository
 
-The Command-Line Task you will need to run for the AP Pipeline is
-``ap_pipe/bin/ap_pipe.py``.
+  - The reference catalogs must be in a directory called `ref_cats` with subdirectories
+    called `gaia` and `pan-starrs` containing the appropriate catalog shards
+
+- **Calibration products** (biases, flats, and defects, if applicable)
+  ingested into a Butler repository you must specify with the ``--calib`` flag on
+  the command line at runtime
+
+  - To check if this requirement has been satisfied, you can inspect the
+    `calibRegistry.sqlite3` created in this repository and ensure the information
+    in the flat, bias, and defect tables is accurate
+
+- **Template images** (of type ``deepCoadd`` by default) for difference imaging
+  must be either in the main Butler repository or in another location you may
+  specify with the ``--template`` flag on the command line at runtime
 
-For example, to process ingested data in ``input_loc`` with calibration products
-(including an appropriate template) residing in ``calib_loc``, one may run
+A sample dataset from the `DECam HiTS survey <http://iopscience.iop.org/article/10.3847/0004-637X/832/2/155/meta>`_ 
+that works with ``ap_pipe`` in the :ref:`ap-verify-datasets` format
+is available as `ap_verify_hits2015 
+<https://github.com/lsst/ap_verify_hits2015>`_. However, this dataset must be
+ingested as described in :ref:`section-ap-pipe-ingesting-data-files`, and the reference
+catalog and defect files must be decompressed and extracted.
 
-.. code-block:: none
-   
-   ap_pipe.py input_loc --calib calib_loc --output output_loc --id visit=123456 ccdnum=12
+.. warning::
 
-This command only processes observations that have a ``dataId`` corresponding to
-visit 123456 and ccdnum 12. The user must specify a ``dataId`` string or
-no data will be processed.
+   `lsst.ap.pipe` has only been used on DECam (and, correspondingly, with
+   `lsst.obs.decam`) so far, and does not yet support data from other cameras.
+   This functionality is coming soon
+   (see `DM-12315 <https://jira.lsstcorp.org/browse/DM-12315>`_).
 
-`lsst.ap.pipe` has only been used on DECam (and, correspondingly, with `lsst.obs.decam`)
-so far, and does not yet support data from other cameras.
+Please continue to :doc:`Pipeline Tutorial <pipeline-tutorial>` for more
+details about running the AP Pipeline and interpreting the results.

doc/lsst.ap.pipe/index.rst

@@ -9,8 +9,8 @@ lsst.ap.pipe
 The ``lsst.ap.pipe`` module links together a set of common image processing
 tasks so that a user may run one Command-Line Task on a dataset of raw,
 ingested images rather than several. The Alert Production (AP) pipeline
-includes four key data processing Tasks for LSST Prompt Data Products:
-`~lsst.ip.isr.IsrTask`, `~lsst.pipe.tasks.ProcessCcdTask`, 
+includes three key data processing Tasks for LSST Prompt Data Products:
+`~lsst.pipe.tasks.ProcessCcdTask` (which includes `~lsst.ip.isr.IsrTask`), 
 `~lsst.ip.diffim.ImageDifferenceTask`, and
 `~lsst.ap.associate.AssociationTask`.
 

doc/lsst.ap.pipe/pipeline-overview.rst

@@ -13,12 +13,22 @@ difference image source catalogs, and a source association database.
 The initial motivation for `lsst.ap.pipe`, information about one of the original test datasets,
 and an outdated tutorial are available in `DMTN-039 <https://dmtn-039.lsst.io>`_.
 
+The AP Pipeline calls three main tasks and their associated subtasks:
+
+1. `~lsst.pipe.tasks.ProcessCcdTask`, which in turn calls `lsst.ip.isr.IsrTask`,
+   `lsst.pipe.tasks.CharacterizeImageTask`, and `lsst.pipe.tasks.CalibrateTask`
+   to perform image reduction as well as photometric and astrometric calibration;
+2. `~lsst.pipe.tasks.ImageDifferenceTask`, which uses many utilities from
+   `lsst.ip.diffim`; and
+3. `~lsst.ap.associate.AssociationTask`, which makes a catalog of
+   Difference Image Analysis (DIA) Objects from the DIASources created
+   during image differencing.
+
 In practice, `lsst.ap.pipe` is often discussed in the context of `lsst.ap.verify`.
 The former is responsible for running the AP Pipeline. The latter uses `lsst.ap.pipe`
 to verify the output.
 
 `ap_pipe` is entirely written in Python. Key contents include:
 
 - `~lsst.ap.pipe.ApPipeTask`: a `~lsst.pipe.base.CmdLineTask` for running the entire AP Pipeline
-- `~lsst.ap.pipe.MaxPsfWcsSelectImagesTask`: a `~lsst.pipe.tasks.selectImages.WcsSelectImagesTask` 
-    for selecting images with good PSFs to create a coadd template
+