Add context for distinguishing Gen 2 and Gen 3.

doc/lsst.ap.pipe/getting-started-gen2.rst

@@ -7,6 +7,12 @@ Getting started with the AP pipeline (Gen 2)
 ############################################
 
 
+This page explains how to set up a Gen 2 data repository that can then be processed with the AP Pipeline (see :doc:`pipeline-tutorial-gen2`).
+This is the established Science Pipelines workflow, and is compatible with a variety of existing pipelines and tools.
+However, it is expected to be phased out in the future in favor of the Gen 3 framework.
+
+If you already have a Gen 3 data repository or want to learn the new framework, see :doc:`getting-started`.
+
 .. _section-ap-pipe-installation-gen2:
 
 Installation

doc/lsst.ap.pipe/getting-started.rst

@@ -8,6 +8,11 @@
 Getting started with the AP pipeline (Gen 3)
 ############################################
 
+This page explains how to set up a Gen 3 data repository that can then be processed with the AP Pipeline (see :doc:`pipeline-tutorial`).
+This is appropriate if you are trying to learn the new workflow, and compatibility or integration with other tools is not a problem.
+The Gen 3 processing is still being finalized, and all details in these tutorials are subject to change.
+
+If you already have a Gen 2 data repository or need compatibility with existing code, see :doc:`getting-started-gen2`.
 
 .. _section-ap-pipe-installation:
 

doc/lsst.ap.pipe/index.rst

@@ -8,8 +8,17 @@ lsst.ap.pipe
 
 .. Paragraph that describes what this Python module does and links to related modules and frameworks.
 
-The ``lsst.ap.pipe`` module links together a set of common image processing tasks so that a user may run one Command-Line Task on a dataset of raw, ingested images rather than several.
-The Alert Production (AP) pipeline includes three key data processing Tasks for LSST Prompt Data Products: `~lsst.pipe.tasks.ProcessCcdTask` (which includes `~lsst.ip.isr.IsrTask`),  `~lsst.ip.diffim.ImageDifferenceTask`, and `~lsst.ap.associate.AssociationTask`.
+The ``lsst.ap.pipe`` module links together a set of common image processing tasks so that a user may run one command on a dataset of raw, ingested images rather than several.
+The Alert Production (AP) pipeline includes the following key data processing Tasks for LSST Prompt Data Products: `~lsst.ip.isr.IsrTask`, `~lsst.pipe.tasks.characterizeImage.CharacterizeImageTask`, `~lsst.pipe.tasks.calibrate.CalibrateTask`, `~lsst.ip.diffim.ImageDifferenceTask`, and `~lsst.ap.association.DiaPipelineTask`.
+
+At present, the alert production pipeline is implemented using two separate frameworks that store and retrieve data from "Butler" repositories in incompatible ways:
+
+- `ApPipeTask` is an `lsst.pipe.base.CmdLineTask` that reads and writes data using the :ref:`lsst.daf.persistence` package.
+  This is the established "Gen 2" framework.
+- ``ApPipe`` is an `lsst.pipe.base.Pipeline` that reads and writes data using the :ref:`lsst.daf.butler` package.
+  This "Gen 3" framework is expected to be the only implementation in the future.
+
+.. TODO: add links to Gen 3 docs as they become available
 
 Overview
 ========

doc/lsst.ap.pipe/pipeline-overview.rst

@@ -7,22 +7,22 @@ Overview of the AP pipeline
 ###########################
 
 :doc:`lsst.ap.pipe <index>` is a data processing pipeline for Prompt Data Products.
-It is a Command-Line Task which operates on ingested raw data in a Butler repository.
+It operates on ingested raw data in a Butler repository.
 It also requires appropriate calibration products and templates. As it runs,
-`ApPipeTask` generates calibrated exposures, difference images,
+the pipeline generates calibrated exposures, difference images,
 difference image source catalogs, and a source association database.
 
 The initial motivation for :doc:`lsst.ap.pipe <index>`, information about one of the original test datasets,
 and an outdated tutorial are available in `DMTN-039 <https://dmtn-039.lsst.io>`_.
 
-The AP Pipeline calls three main tasks and their associated subtasks:
+The AP Pipeline calls several main tasks and their associated subtasks:
 
-1. `~lsst.pipe.tasks.ProcessCcdTask`, which in turn calls `lsst.ip.isr.IsrTask`,
-   `lsst.pipe.tasks.CharacterizeImageTask`, and `lsst.pipe.tasks.CalibrateTask`
-   to perform image reduction as well as photometric and astrometric calibration;
-2. `~lsst.pipe.tasks.ImageDifferenceTask`, which uses many utilities from
+#. `~lsst.ip.isr.IsrTask`, which performs image reduction;
+#. `~lsst.pipe.tasks.CharacterizeImageTask`, which estimates the background and point-spread function of an image;
+#. `~lsst.pipe.tasks.CalibrateTask`, which performs photometric and astrometric calibration;
+#. `~lsst.pipe.tasks.ImageDifferenceTask`, which uses many utilities from
    :doc:`lsst.ip.diffim </modules/lsst.ip.diffim/index>`; and
-3. `~lsst.ap.associate.AssociationTask`, which makes a catalog of
+#. `~lsst.ap.association.DiaPipelineTask`, which makes a catalog of
    Difference Image Analysis (DIA) Objects from the DIASources created
    during image differencing.
 
@@ -32,7 +32,7 @@ to verify the output.
 
 :doc:`ap_pipe <index>` is entirely written in Python. Key contents include:
 
-- `ApPipeTask`: a `~lsst.pipe.base.CmdLineTask` for running the entire AP Pipeline
+- `ApPipeTask`: a `~lsst.pipe.base.CmdLineTask` for running the entire AP Pipeline in the older, "Gen 2" data processing framework
 - `ApPipeConfig`: a config for customizing ``ApPipeTask`` for a particular dataset's needs.
   Supported observatory packages should provide a :ref:`config override file <command-line-task-config-howto-obs>` that does most of the work.
-
+- :file:`ApPipe.yaml`: a `~lsst.pipe.base.Pipeline` configuration for running the entire AP Pipeline in the newer, "Gen 3" framework