DRP.yaml

description: |
  The full DRP pipeline for large productions that include data
  taken with the HSC-I2 and HSC-R2 bands. First used for the 2023 internal
  PDR2 reprocessing at the USDF. All tasks that can be split by data query are
  grouped together in subsets.
instrument: lsst.obs.subaru.HyperSuprimeCam
imports:
  - location: $DRP_PIPE_DIR/pipelines/HSC/DRP-PDR4.yaml

subsets:
  # steps 1, 2a, 2b, 2c, 2d, 2e, and 7 are the same as DRP-PDR4.yaml
  step3:
    subset:
      - makeWarp
      - assembleCoadd
      - detection
      - mergeDetections
      - deblend
      - measure
      - mergeMeasurements
      - forcedPhotCoadd
      - transformObjectTable
      - writeObjectTable
      - consolidateObjectTable
      - healSparsePropertyMaps
      - selectGoodSeeingVisits
      - templateGen
    description: |
      Tasks that can be run together, but only after the 'step1' and 'step2'
      subsets.

      These should be run with explicit 'tract' constraints essentially all the
      time, because otherwise quanta will be created for jobs with only partial
      visit coverage.

      This subset is considered a workaround for missing middleware and task
      functionality.  It may be removed in the future.
  step4:
    subset:
      - forcedPhotCcd
      - forcedPhotDiffim
      - getTemplate
      - imageDifference
      - transformDiaSourceCat
      - writeForcedSourceTable
    description: |
      Tasks that can be run together, but only after the 'step1', 'step2' and
      'step3' subsets

      These detector-level tasks should not be run with 'tract' or 'patch' as
      part of the data ID expression if all reference catalogs or diffIm
      templates that cover these detector-level quanta are desired.
  step5:
    subset:
      - drpAssociation
      - drpDiaCalculation
      - forcedPhotCcdOnDiaObjects
      - forcedPhotDiffOnDiaObjects
      - transformForcedSourceTable
      - consolidateAssocDiaSourceTable
      - consolidateFullDiaObjectTable
      - writeForcedSourceOnDiaObjectTable
      - transformForcedSourceOnDiaObjectTable
    description: |
      Tasks that can be run together, but only after the 'step1', 'step2',
      'step3', and 'step4' subsets

      This step includes patch-level aggregation Tasks. These should be run
      with explicit 'tract' constraints in the data query, otherwise quanta
      will be created for jobs with only partial visit coverage.
  step6:
    subset:
      - consolidateDiaSourceTable
    description: |
      Tasks that can be run together, but only after the 'step1', 'step2',
      'step3', and 'step4' subsets

      This step includes visit-level aggregation tasks. Running without tract
      or patch in the data query is recommended, otherwise the outputs of
      consolidateDiaSourceTable will not contain complete visits.

      This subset is separate from step4 to signal to operators to pause to
      assess unexpected image differencing failures before these aggregation
      steps. Otherwise, if run in the same quantum graph, aggregated data
      products (e.g. diaObjects) would not be created if one or more of the
      expected inputs is missing.