DRP-ops-rehearsal-3.yaml

description: DRP Pipeline for Simulated ComCam Ops Rehearsal 3
instrument: lsst.obs.lsst.LsstComCamSim
imports:
  - $DRP_PIPE_DIR/pipelines/_ingredients/LSSTComCamSim/DRP.yaml

subsets:
  step1:
    subset:
      - isr
      - characterizeImage
      - calibrate
      - writePreSourceTable
      - transformPreSourceTable
    description: |
      Per-detector tasks that can be run together to start the DRP pipeline.

      These should never be run with 'tract' or 'patch' as part of the data ID
      expression if any later steps will also be run, because downstream steps
      require full visits and 'tract' and 'patch' constraints will always
      select partial visits that overlap that region.
  step2a:
    subset:
      - analyzeMatchedPreVisitCore
      - consolidatePreSourceTable
      - consolidateVisitSummary
    description: |
      Visit-level tasks
      Allowed data query constraints: visit

      Tasks aggregate all detectors for a given visit and perform
      full focal plane background estimation.

      TODO: Evaluate SkyCorr after intial DRP
  step2b:
    subset:
      - isolatedStarAssociation
      - gbdesAstrometricFit
    description: |
      Tract-level tasks
      Allowed data query constraints: tract

      GbdesAstrometricFit and isolatedStarAssociation both use PreSources,
      generated by consolidatePreSourceTable, for all visits that overlap a
      tract.
      gbdesAstrometricFit produces solutions per-tract, per-visit
      isolatedStarAssociation produces solutions per-tract.
      TODO: Evaluate GBDES after first DRP
  step2c:
    subset:
      - fgcmBuildFromIsolatedStars
      - fgcmFitCycle
      - fgcmOutputProducts
  step2d:
    subset:
      - finalizeCharacterization
      - writeRecalibratedSourceTable
      - transformSourceTable
      - consolidateSourceTable
      - updateVisitSummary
    description: |
      Visit-level tasks.
      Allowed data query constraints: visit

      writeRecalibratedSourceTable, transformSourceTable run per-detector
      consolidateSourceTable produces one data product per visit.
      finalizeCharacterization will eventually model full focal plane PSFs.
  step2e:
    subset:
      - makeCcdVisitTable
      - makeVisitTable
    description: |
      Global-level tasks that must not be run with any data query constraints
      Can be run anytime after subset step2d.

      Allowed data query constraints: instrument

      Tasks generate one data product per collection.
      make[Ccd]VisitTable produces per-collection summary of the Visits
      and CcdVisits.
  step3:
    subset:
      - makeWarp
      - selectDeepCoaddVisits
      - assembleCoadd
      - detection
      - mergeDetections
      - deblend
      - measure
      - mergeMeasurements
      - forcedPhotCoadd
      - transformObjectTable
      - writeObjectTable
      - consolidateObjectTable
      - healSparsePropertyMaps
      - selectGoodSeeingVisits
      - templateGen
      # analysis_tools tasks
      - analyzeMatchedVisitCore
      - analyzeObjectTableCore
      - catalogMatchTract
      - photometricCatalogMatch
      - photometricRefCatObjectTract
      - plotPropertyMapTract
      - refCatObjectTract
      - validateObjectTableCore
    description: |
      Tasks that can be run together, but only after the 'step1' and 'step2d'
      subsets.

      These should be run with explicit 'tract' constraints essentially all the
      time, because otherwise quanta will be created for jobs with only partial
      visit coverage.

      This subset is considered a workaround for missing middleware and task
      functionality.  It may be removed in the future.
  step4:
    subset:
      - forcedPhotCcd
      - forcedPhotDiffim
      - getTemplate
      - subtractImages
      - detectAndMeasureDiaSources
      - transformDiaSourceCat
      - writeForcedSourceTable
    description: |
      Tasks that can be run together, but only after the 'step1', 'step2d' and
      'step3' subsets

      These detector-level tasks should not be run with 'tract' or 'patch' as
      part of the data ID expression if all reference catalogs or diffIm
      templates that cover these detector-level quanta are desired.
  step5:
    subset:
      - drpAssociation
      - drpDiaCalculation
      - forcedPhotCcdOnDiaObjects
      - forcedPhotDiffOnDiaObjects
      - transformForcedSourceTable
      - consolidateAssocDiaSourceTable
      - consolidateFullDiaObjectTable
      - writeForcedSourceOnDiaObjectTable
      - transformForcedSourceOnDiaObjectTable
      - analyzeDiaSourceTableTract
    description: |
      Tasks that can be run together, but only after the 'step1', 'step2',
      'step3', and 'step4' subsets

      This step includes patch-level aggregation Tasks. These should be run
      with explicit 'tract' constraints in the data query, otherwise quanta
      will be created for jobs with only partial visit coverage.
  step6:
    subset:
      - consolidateDiaSourceTable
    description: |
      Tasks that can be run together, but only after the 'step1', 'step2',
      'step3', and 'step4' subsets

      This step includes visit-level aggregation tasks. Running without tract
      or patch in the data query is recommended, otherwise the outputs of
      consolidateDiaSourceTable will not contain complete visits.

      This subset is separate from step4 to signal to operators to pause to
      assess unexpected image differencing failures before these aggregation
      steps. Otherwise, if run in the same quantum graph, aggregated data
      products (e.g. diaObjects) would not be created if one or more of the
      expected inputs is missing.
  step7:
    subset:
      - consolidateHealSparsePropertyMaps
      - analyzeObjectTableSurveyCore
    description: |
      Tasks that should be run as the final step that require global inputs,
      and can be run after the 'step3' subset.

      This step has global aggregation tasks to run over all visits, detectors,
      tracts, etc.  This step should be run only with the instrument constraint
      in the data query.