DM-23436: apdb creation instructions outdated

README.md

@@ -12,9 +12,3 @@ The user must specify the main repository with ingested images (and the
 location of the calibration products and templates if they reside elsewhere),
 the name of the association database (may be either created from scratch or
 connected to for continued associating), and a Butler data ID.
-
-## Example command-line usage
-
-```
-ap_pipe.py repo --calib repo/calibs --rerun processed -c associator.level1_db.db_name=apdb/association.db --id visit=123456 ccdnum=42 filter=g --template templates
-```

doc/lsst.ap.pipe/apdb.rst

@@ -14,22 +14,23 @@ Setting up the Alert Production Database for ap_pipe
 
 .. |ap_pipe| replace:: :command:`ap_pipe.py`
 
-|ap_pipe| needs an existing Alert Production Database (APDB) in which to store its results.
-Such a database will be provided externally during operations, but developers can run |make_apdb| to set up their own database for testing.
+
+In its default configuration, the Alert Production Pipeline, as represented by :lsst-task:`lsst.ap.pipe.ApPipeTask` and executed by |ap_pipe|, relies on a database to save and load DIASources and DIAObjects.
+When running as part of the operational system, this database will be provided externally.
+However, during testing and development, developers can run |make_apdb| to set up their own database.
 This page provides an overview of how to use |make_apdb|.
 
 .. _section-ap-pipe-apdb-config:
 
 Configuring the database
 ========================
 
-|ap_pipe| includes information about the database location and schema in its `ApPipeConfig`, and most users pass this information to the script through the :option:`--config <ap_pipe.py --config>` and :option:`--configfile <ap_pipe.py --configfile>` command-line options.
-
+The database is configured using `~lsst.dax.apdb.ApdbConfig`.
+|ap_pipe| command line users can pass configuration information to the script through the :option:`--config <ap_pipe.py --config>` and :option:`--configfile <ap_pipe.py --configfile>` command-line options.
 |make_apdb| also uses `ApPipeConfig` and the :option:`--config` and :option:`--configfile` options, so users can pass exactly the same arguments to |make_apdb| and |ap_pipe|.
 Supporting identical command line arguments for both scripts makes it easy to keep the database settings in sync.
 
-For more information on the configuration options themselves, see `lsst.dax.apdb.ApdbConfig`.
-``apdb.db_url`` has no default and must be set to create a valid config.
+Note that ``apdb.db_url`` has no default; a value *must* be provided by the user.
 
 .. _section-ap-pipe-apdb-examples:
 
@@ -40,8 +41,8 @@ Databases can be configured using direct config overrides:
 
 .. prompt:: bash
 
-   make_apdb.py -c apdb.isolation_level=READ_UNCOMMITTED apdb.db_url="sqlite:///databases/apdb.db" differencer.coaddName=dcr
-   ap_pipe.py -c apdb.isolation_level=READ_UNCOMMITTED apdb.db_url="sqlite:///databases/apdb.db" differencer.coaddName=dcr repo --calib repo/calibs --rerun myrun --id
+   make_apdb.py -c diaPipe.apdb.isolation_level=READ_UNCOMMITTED diaPipe.apdb.db_url="sqlite:///databases/apdb.db" differencer.coaddName=dcr
+   ap_pipe.py -c diaPipe.apdb.isolation_level=READ_UNCOMMITTED diaPipe.apdb.db_url="sqlite:///databases/apdb.db" differencer.coaddName=dcr repo --calib repo/calibs --rerun myrun --id
 
 |make_apdb| ignores any `ApPipeConfig` fields not related to the APDB (in the example, ``differencer.coaddName``), so there is no need to filter them out.
 

doc/lsst.ap.pipe/getting-started.rst

@@ -47,7 +47,7 @@ A standard ingestion workflow for DECam looks something like
 
 .. note::
 
-   Defect ingestion is a step unique to DECam. It presently requires 
+   Defect ingestion is a step unique to DECam. It presently requires
    ``--mode=skip``, this mode interprets paths as relative to ``calib_loc``,
    and the validity value is not used (but must be included). This interface
    may change when `DM-5467 <https://jira.lsstcorp.org/browse/DM-5467>`_ is completed.
@@ -66,17 +66,17 @@ For the AP Pipeline to successfully process data, the following is required:
 
   - The reference catalogs must be in a directory called :file:`ref_cats` with subdirectories
     for each catalog containing the appropriate catalog shards.
-    We recommend using Pan-STARRS for photometry and gaia for astrometry.
+    We recommend using Pan-STARRS for photometry and Gaia for astrometry.
     An example :ref:`config file <command-line-task-config-howto-configfile>` for using these two catalogs can be found in the `ap_verify_hits2015`_ repository.
-    
+
 - **Calibration products** (biases, flats, and defects, if applicable)
   ingested into a Butler repository you must specify with the ``--calib`` flag on
   the command line at runtime
-  
+
   - To check if this requirement has been satisfied, you can inspect the
     :file:`calibRegistry.sqlite3` created in this repository and ensure the information
     in the flat, bias, and defect tables is accurate
-    
+
 - **Template images** (of type ``deepCoadd`` by default) for difference imaging
   must be either in the main Butler repository or in another location you may
   specify with the ``--template`` flag on the command line at runtime

doc/lsst.ap.pipe/index.rst

@@ -16,7 +16,7 @@ Overview
 
 .. toctree::
    :maxdepth: 1
-   
+
    pipeline-overview
 
 .. .. _lsst.ap.pipe-using:
@@ -26,7 +26,7 @@ Using lsst.ap.pipe
 
 .. toctree::
    :maxdepth: 1
-   
+
    getting-started
    pipeline-tutorial
    apdb
@@ -49,6 +49,18 @@ Script reference
 
    scripts/make_apdb.py
 
+Task reference
+==============
+
+.. _lsst.ap.pipe-command-line-tasks:
+
+Command-line tasks
+------------------
+
+.. lsst-cmdlinetasks::
+   :root: lsst.ap.pipe
+   :toctree: tasks
+
 .. _lsst.ap.pipe-pyapi:
 
 Python API reference

doc/lsst.ap.pipe/pipeline-tutorial.rst

@@ -13,7 +13,7 @@ Pick up where you left off in :doc:`Getting Started <getting-started>`.
 This means you already have a repository of ingested DECam data and have setup
 the LSST Science Pipelines stack as well as ``ap_pipe`` and ``ap_association``.
 
-Your directory structure should look something like 
+Your directory structure should look something like
 
 .. code-block:: none
 
@@ -66,8 +66,8 @@ To process your ingested data, run
 .. prompt:: bash
 
    mkdir apdb/
-   make_apdb.py -c apdb.isolation_level=READ_UNCOMMITTED -c apdb.db_url="sqlite:///apdb/association.db"
-   ap_pipe.py repo --calib repo/calibs --rerun processed -c apdb.isolation_level=READ_UNCOMMITTED -c apdb.db_url="sqlite:///apdb/association.db" --id visit=123456 ccdnum=42 filter=g --template templates
+   make_apdb.py -c diaPipe.apdb.isolation_level=READ_UNCOMMITTED -c diaPipe.apdb.db_url="sqlite:///apdb/association.db"
+   ap_pipe.py repo --calib repo/calibs --rerun processed -c diaPipe.apdb.isolation_level=READ_UNCOMMITTED -c diaPipe.apdb.db_url="sqlite:///apdb/association.db" --id visit=123456 ccdnum=42 filter=g --template templates
 
 In this case, a ``processed`` directory will be created within ``repo/rerun`` and the results will be written there.
 See :doc:`apdb` for more information on :command:`make_apdb.py`.
@@ -89,7 +89,7 @@ If you prefer to have a standalone output repository, you may instead run
 
 .. prompt:: bash
 
-   ap_pipe.py repo --calib repo/calibs --output path/to/put/processed/data/in -c apdb.isolation_level=READ_UNCOMMITTED -c apdb.db_url="sqlite:///apdb/association.db" --id visit=123456 ccdnum=42 filter=g --template path/to/templates
+   ap_pipe.py repo --calib repo/calibs --output path/to/put/processed/data/in -c diaPipe.apdb.isolation_level=READ_UNCOMMITTED -c diaPipe.apdb.db_url="sqlite:///apdb/association.db" --id visit=123456 ccdnum=42 filter=g --template path/to/templates
 
 In this case, the output directory will be created if it does not already exist.
 If you omit the ``--template`` flag, ``ap_pipe`` will assume the templates are
@@ -130,7 +130,7 @@ in the ``processed`` directory and access the data products that way.
 For example, in python
 
 .. code-block:: python
-   
+
    import lsst.daf.persistence as dafPersist
    butler = dafPersist.Butler('repo/rerun/processed')
    dataId = {'visit': 123456, 'ccdnum': 42, 'filter': 'g'}
@@ -149,7 +149,7 @@ as templates for difference imaging. However, the pipeline also supports
 using calibrated exposures (``calexps``) as templates instead. A configuration file
 ``config/calexpTemplates.py`` is included witha ``ap_pipe`` to enable this.
 
-To use ap_pipe in calexp template mode, point to the config file with the 
+To use ap_pipe in calexp template mode, point to the config file with the
 ``--configfile`` (``-C``) flag and additionally specify the ``dataId`` of the template
 with the ``--templateId`` flag, e.g.,
 
@@ -162,8 +162,8 @@ with the ``--template`` flag if they are not in the main repo.
 A full command looks like
 
 .. prompt:: bash
-   
-   ap_pipe.py repo --calib repo/calibs --rerun processed -C $AP_PIPE_DIR/config/calexpTemplates.py -c apdb.isolation_level=READ_UNCOMMITTED -c apdb.db_url="sqlite:///apdb/association.db" --id visit=123456 ccdnum=42 filter=g --template /path/to/calexp/templates --templateId visit=234567
+
+   ap_pipe.py repo --calib repo/calibs --rerun processed -C $AP_PIPE_DIR/config/calexpTemplates.py -c diaPipe.apdb.isolation_level=READ_UNCOMMITTED -c diaPipe.apdb.db_url="sqlite:///apdb/association.db" --id visit=123456 ccdnum=42 filter=g --template /path/to/calexp/templates --templateId visit=234567
 
 
 .. _section-ap-pipe-supplemental-info:
@@ -211,8 +211,8 @@ Common errors
 Interpreting the results
 ========================
 
-.. warning:: 
-   
+.. warning::
+
    The format of the ``ap_association`` Alert Production Database is rapidly evolving. For
    the latest information on how to interface with it, see :doc:`lsst.ap.association </modules/lsst.ap.association/index>`.
 
@@ -241,7 +241,7 @@ calibrated exposures, difference images, inspect DIAObjects and/or DIASources, e
    objTable = pd.read_sql_query('select * from {0};'.format(tables['obj']), connection)
    srcTable = pd.read_sql_query('select * from {0};'.format(tables['src']), connection)
    connection.close()
-   
+
    # Plot how many sourceIDs are attached to any given objectID
    obj_id = objTable['id'].values  # object ids from the objTable
    con_obj_id = conTable['obj_id'].values  # object ids from the conTable

doc/lsst.ap.pipe/tasks/lsst.ap.pipe.ApPipeTask.rst

@@ -0,0 +1,44 @@
+.. lsst-task-topic:: lsst.ap.pipe.ApPipeTask
+
+##########
+ApPipeTask
+##########
+
+``ApPipeTask`` represents the alert production pipeline.
+``ApPipeTask`` is available as a `command-line task <pipe-tasks-command-line-tasks>`, ``ap_pipe.py``.
+
+.. .. _lsst.ap.pipe.ApPipeTask-summary:
+..
+.. Processing summary
+.. ==================
+
+.. .. _lsst.ap.pipe.ApPipeTask-cli:
+..
+.. ap_pipe.py command-line interface
+.. =================================
+
+.. _lsst.ap.pipe.ApPipeTask-api:
+
+Python API summary
+==================
+
+.. lsst-task-api-summary:: lsst.ap.pipe.ApPipeTask
+
+.. .. _lsst.ap.pipe.ApPipeTask-butler:
+..
+.. Butler datasets
+.. ===============
+
+.. _lsst.ap.pipe.ApPipeTask-subtasks:
+
+Retargetable subtasks
+=====================
+
+.. lsst-task-config-subtasks:: lsst.ap.pipe.ApPipeTask
+
+.. _lsst.ap.pipe.ApPipeTask-configs:
+
+Configuration fields
+====================
+
+.. lsst-task-config-fields:: lsst.ap.pipe.ApPipeTask