DM-16535: Implement MetricRegistry

doc/lsst.verify/index.rst

@@ -14,6 +14,8 @@ Measurements made through `lsst.verify` can be uploaded to LSST's SQUASH_ monito
 
    `SQR-019: LSST Verification Framework API Demonstration <https://sqr-019.lsst.io>`_.
 
+.. _lsst.verify-using:
+
 Using lsst.verify
 =================
 
@@ -22,6 +24,39 @@ Using lsst.verify
 
    inspect_job
 
+.. _lsst.verify-command-line-taskref:
+
+Task reference
+==============
+
+.. .. _lsst.verify-command-line-tasks:
+..
+.. Command-line tasks
+.. ------------------
+..
+.. .. lsst-cmdlinetasks::
+..    :root: lsst.verify
+
+.. _lsst.verify-tasks:
+
+Tasks
+-----
+
+.. lsst-tasks::
+   :root: lsst.verify
+   :toctree: tasks
+
+.. .. _lsst.verify-configs:
+..
+.. Configurations
+.. --------------
+..
+.. .. lsst-configs::
+..    :root: lsst.verify
+..    :toctree: configs
+
+.. _lsst.verify-pyapi:
+
 Python API reference
 ====================
 

doc/lsst.verify/tasks/lsst.verify.compatibility.MetricTask.rst

@@ -0,0 +1,60 @@
+.. lsst-task-topic:: lsst.verify.compatibility.MetricTask
+
+.. currentmodule:: lsst.verify.compatibility
+
+######################################
+MetricTask (lsst.verify.compatibility)
+######################################
+
+``MetricTask`` is a base class for generating `lsst.verify.Measurement` given input data.
+Each ``MetricTask`` class accepts specific type(s) of datasets and produces measurements for a specific metric or family of metrics.
+
+While its API mirrors that of `~lsst.pipe.base.PipelineTask`, this version of ``MetricTask`` is designed to be used with the Gen 2 framework.
+
+.. _lsst.verify.compatibility.MetricTask-api:
+
+Python API summary
+==================
+
+.. lsst-task-api-summary:: lsst.verify.compatibility.MetricTask
+
+.. _lsst.verify.compatibility.MetricTask-subtasks:
+
+Retargetable subtasks
+=====================
+
+.. lsst-task-config-subtasks:: lsst.verify.compatibility.MetricTask
+
+.. _lsst.verify.compatibility.MetricTask-configs:
+
+Configuration fields
+====================
+
+.. lsst-task-config-fields:: lsst.verify.compatibility.MetricTask
+
+.. _lsst.verify.compatibility.MetricTask-indepth:
+
+In Depth
+========
+
+.. _lsst.verify.compatibility.MetricTask-indepth-subclassing:
+
+Subclassing
+-----------
+
+``MetricTask`` is primarily customized using the `~MetricTask.run` or `~MetricTask.adaptArgsAndRun` methods.
+Each subclass must also implement the `~MetricTask.getOutputMetricName` method.
+
+The task config should use `lsst.pipe.base.InputDatasetConfig` to identify input datasets as if it were a `~lsst.pipe.base.PipelineTask`.
+Only the ``name`` field is used in a Gen 2 context, but use of `~lsst.pipe.base.InputDatasetConfig` is expected to simplify the transition to Gen 3.
+
+.. _lsst.verify.compatibility.MetricTask-indepth-register:
+
+Registration
+------------
+
+The most common way to run ``MetricTask`` is as plugins to :lsst-task:`~lsst.verify.compatibility.MetricsControllerTask`.
+Most ``MetricTask`` classes should use the `register` decorator to assign a plugin name.
+
+Because of implementation limitations, each registered name may appear at most once in `MetricsControllerConfig`.
+If you expect to need multiple instances of the same ``MetricTask`` class (typically when the same class can compute multiple metrics), it must have the `registerMultiple` decorator instead.

doc/lsst.verify/tasks/lsst.verify.compatibility.MetricsControllerTask.rst

@@ -0,0 +1,109 @@
+.. lsst-task-topic:: lsst.verify.compatibility.MetricsControllerTask
+
+.. currentmodule:: lsst.verify.compatibility
+
+#####################
+MetricsControllerTask
+#####################
+
+``MetricsControllerTask`` runs collections of :lsst-task:`lsst.verify.compatibility.MetricTask`, and stores the resulting `~lsst.verify.Measurement` objects using the `~lsst.verify.Job` persistence framework.
+It is a stand-in for functionality provided by the Gen 3 Tasks framework.
+The datasets that ``MetricsControllerTask`` consumes depend on the :lsst-task:`~lsst.verify.compatibility.MetricTask`\ s to be run, and are handled automatically.
+
+``MetricsControllerTask`` is not a command-line task, but may be called from within both task- and non-task pipelines.
+
+.. _lsst.verify.compatibility.MetricsControllerTask-summary:
+
+Processing summary
+==================
+
+Unlike most tasks, ``MetricsControllerTask`` has a `~MetricsControllerTask.runDataRefs` method that takes a list of data references.
+``MetricsControllerTask`` calls every :lsst-task:`~lsst.verify.compatibility.MetricTask` in :lsst-config-field:`~lsst.verify.compatibility.MetricsControllerConfig.measurers` on every data reference, loading any datasets necessary.
+It produces one `~lsst.verify.Job` object for each input data reference, and writes them to disk.
+
+.. _lsst.verify.compatibility.MetricsControllerTask-api:
+
+Python API summary
+==================
+
+.. lsst-task-api-summary:: lsst.verify.compatibility.MetricsControllerTask
+
+.. _lsst.verify.compatibility.MetricsControllerTask-subtasks:
+
+Retargetable subtasks
+=====================
+
+.. lsst-task-config-subtasks:: lsst.verify.compatibility.MetricsControllerTask
+
+.. _lsst.verify.compatibility.MetricsControllerTask-configs:
+
+Configuration fields
+====================
+
+.. lsst-task-config-fields:: lsst.verify.compatibility.MetricsControllerTask
+
+.. _lsst.verify.compatibility.MetricsControllerTask-indepth:
+
+In Depth
+========
+
+Because ``MetricsControllerTask`` applies every :lsst-task:`~lsst.verify.compatibility.MetricTask` to every input data reference indiscriminately, it may not give good results with metrics or data references having a mixture of granularities (e.g., CCD-level, visit-level, dataset-level).
+The recommended way around this limitation is to create multiple ``MetricsControllerTask`` objects, and configure each one for metrics of a single granularity.
+
+Each :lsst-task:`~lsst.verify.compatibility.MetricTask` in a ``MetricsControllerTask`` must measure a different metric, or they will overwrite each others' values.
+
+.. _lsst.verify.compatibility.MetricsControllerTask-examples:
+
+Examples
+========
+
+Typically, a user of ``MetricsControllerTask`` will configure it with tasks that have `register` decorator:
+
+.. code-block:: py
+
+   from lsst.verify.compatibility import register, \
+       MetricTask, MetricsControllerTask
+
+
+   @register("ultimate")
+   class UltimateMetric(MetricTask):
+       ...
+
+
+   @register("second")
+   class SecondaryMetric(MetricTask):
+       ...
+
+
+   config = MetricsControllerTask.ConfigClass()
+   config.measurers = ["ultimate", "second"]
+   config.measurers["ultimate"].answer = 42
+   task = MetricsControllerTask(config)
+
+   # CCD-level metrics need CCD-level datarefs
+   # Exact dataset type doesn't matter
+   datarefs = [butler.subset("calexp", visit=42, ccd=ccd)
+               for ccd in range(1, 101)]
+   struct = task.runDataRefs(datarefs)
+   assert len(struct.jobs) == len(datarefs)
+
+A :lsst-task:`~lsst.verify.compatibility.MetricTask` must have the `registerMultiple` decorator to be used multiple times:
+
+.. code-block:: py
+
+   from lsst.verify.compatibility import registerMultiple, MetricTask, MetricsControllerTask
+
+   @registerMultiple("common")
+   class CommonMetric(MetricTask):
+       ...
+
+   config = MetricsControllerTask.ConfigClass()
+   config.measurers = ["common"]
+   config.measurers["common"].configs["case1"] = CommonMetric.ConfigClass()
+   config.measurers["common"].configs["case1"].metric = "misc_tasks.Case1Metric"
+   config.measurers["common"].configs["case2"] = CommonMetric.ConfigClass()
+   config.measurers["common"].configs["case2"].metric = "misc_tasks.Case2Metric"
+   task = MetricsControllerTask(config)
+
+``MetricsControllerTask`` will create and run two different ``CommonMetric`` objects, one configured with ``metric = "misc_tasks.Case1Metric"`` and one with ``metric = "misc_tasks.Case2Metric"``.
+The names ``"case1"`` and ``"case2"`` can be anything, so long as they are unique.

doc/lsst.verify/tasks/lsst.verify.compatibility.SquashMetadataTask.rst

@@ -0,0 +1,39 @@
+.. lsst-task-topic:: lsst.verify.compatibility.metadataTask.SquashMetadataTask
+
+##################
+SquashMetadataTask
+##################
+
+``SquashMetadataTask`` modifies a `~lsst.verify.Job` object to contain metadata that is required for SQuaSH upload.
+It is both the default for the :lsst-config-field:`lsst.verify.compatibility.MetricsControllerConfig.metadataAdder` subtask and its reference implementation.
+
+.. _lsst.verify.compatibility.SquashMetadataTask-summary:
+
+Processing summary
+==================
+
+``SquashMetadataTask`` currently adds the following metadata:
+
+* ``"instrument"``: the name of the instrument that took the data
+* the individual keys of the provided data ID (each `~lsst.verify.Job` produced by :lsst-task:`~lsst.verify.compatibility.MetricsControllerTask` is associated with one data ID, which may be partial)
+
+.. _lsst.verify.compatibility.SquashMetadataTask-api:
+
+Python API summary
+==================
+
+.. lsst-task-api-summary:: lsst.verify.compatibility.SquashMetadataTask
+
+.. _lsst.verify.compatibility.SquashMetadataTask-subtasks:
+
+Retargetable subtasks
+=====================
+
+.. lsst-task-config-subtasks:: lsst.verify.compatibility.SquashMetadataTask
+
+.. _lsst.verify.compatibility.SquashMetadataTask-configs:
+
+Configuration fields
+====================
+
+.. lsst-task-config-fields:: lsst.verify.compatibility.SquashMetadataTask

python/lsst/verify/compatibility/__init__.py

@@ -22,3 +22,4 @@
 from .metricTask import *
 from .metricsControllerTask import *
 from .metadataTask import *
+from .metricRegistry import *