LDM-534.tex

\documentclass[DM,obsolete,STS,toc]{lsstdoc}
\usepackage{enumitem}
\input meta.tex

\begin{document}

\def\product{LSST Level 2 System}

\setDocCompact{true}

\title[Test Spec for \product]{\product~Test Specification}

\author{Jim Bosch, Hsin-Fang Chiang, Michelle Gower, John D. Swinbank}
\setDocRef{\lsstDocType-\lsstDocNum}
\setDocDate{\vcsdate}

\setDocAbstract {
This document describes the detailed test specification for the \product{}.
}

% Most recent last
\setDocChangeRecord{%
	\addtohist{1.0}{2017-06-30}{Initial release of draft version.}{Swinbank}
	\addtohist{2.0}{2017-07-03}{Updated metadata; no content changes.}{Swinbank}
  \addtohist{3.0}{2018-01-11}{Updated to cover the LDM-503-2 (``HSC Reprocessing'') milestone.}{Bosch, Chiang, Gower, Swinbank}
	\addtohist{3.1}{2018-01-11}{Approved in \jira{RFC-425}}{T.~Jenness}
        \addtohist{4.0} {2021-01-06}{Document set to obsolete (\jira{RFC-749}). Test cases moved to \citeds{LDM-639}.}{G. Comoretto}
}

\setDocCurator{John D. Swinbank}
\setDocUpstreamLocation{\url{https://github.com/lsst/ldm-534}}
\setDocUpstreamVersion{\vcsrevision}

\maketitle

\section{Introduction}
\label{sec:intro}

This document specifies the test procedure for the \product{}.

The \product{} is the component of the LSST system which is responsible for
scientific processing leading to:

\begin{itemize}

  \item{Annual data release production;}
  \item{Periodic (re-) generation of calibration products;}
  \item{Periodic (re-) generation of templates for generating difference
  images, to be consumed in the L1 system;}
  \item{Generating QC metrics based on pipeline execution and post-processing of
  scientific data products.}

\end{itemize}

A full description of this product is provided in \S\S7 (which describes the
Data Facility provided execution services) and 13.1 (the science payloads) of
\citeds{LDM-148}.

\subsection{Objectives}
\label{sec:objectives}

This document builds on the description of LSST Data Management's approach to
testing as described in \citeds{LDM-503} to describe the detailed tests that
will be performed on the \product{} as part of the verification of the DM system.

It identifies test designs, test cases and procedures for the tests, and the
pass/fail criteria for each test.

\subsection{Scope}
\label{sec:scope}

This document describes the test procedures for the following components of
the LSST system (as described in \citeds{LDM-148}):

\begin{itemize}

  \item{Services provided by the LSST Data Facility:

    \begin{itemize}
      \item{Template and Calibration Products Production Execution}
      \item{Data Release Production Execution}
      \item{Level 2 Quality Control}
    \end{itemize}
  }

  \item{Science payloads:

    \begin{itemize}
      \item{Annual Calibration Products Production Payload}
      \item{Data Release Production Payload}
      \item{Periodic Calibration Products Production Payload}
      \item{Template Generation Payload}
    \end{itemize}

  }

\end{itemize}

\subsection{Applicable Documents}
\label{sec:docs}

\addtocounter{table}{-1}

\begin{tabular}[htb]{l l}
\citeds{LDM-148} & LSST DM System Architecture \\
\citeds{LDM-151} & LSST DM Science Pipelines Design \\
\citeds{LDM-294} & LSST DM Organization \& Management \\
\citeds{LDM-502} & The Measurement and Verification of DM Key Performance Metrics \\
\citeds{LDM-503} & LSST DM Test Plan \\
\citeds{LSE-61}  & LSST DM Subsystem Requirements \\
\citeds{LSE-163} & LSST Data Products Definition Document \\
\citeds{LSE-180} & Level 2 Photometric Calibration for the LSST Survey \\
\end{tabular}

\subsection{References\label{sect:references}}
\renewcommand{\refname}{}
\bibliography{lsst,refs,books,refs_ads}

%\subsection{Definitions, acronyms, and abbreviations \label{sect:acronyms}} % include acronyms.tex generated by the acronyms.csh (GaiaTools)
%\input{acronyms}


%----------------------------------------------------
% TASK IDENTIFICATION - APPROACH
%----------------------------------------------------
\section{Approach}
\label{sec:approach}

The major activities to be performed are to:

\begin{itemize}

  \item{Compare the design of the Data Release Production payload as
  implemented to the requirements on the outputs of the DM Subsystem as
  defined in \citeds{LSE-63} and \citeds{LSE-163} to demonstrate that all data
  products required by the scientific community will be delivered by the
  system as built.}

  \item{Ensure that all data products included in the DRP payload design are
  correctly produced and persisted appropriately to the LSST Data Backbone
  when executing a data release production.}

  \item{Compare the design of the Calibration Products payloads as implemented
  to the requirements laid down in \citeds{LSE-63}, the overall design
  described in \citeds{LSE-180} and the inputs of the scientific pipeline
  payloads as described in \citeds{LDM-151}.}

  \item{Ensure that all data products included in the CPP payload design are
  correctly produced and persisted appropriately to the LSST Data Backbone
  and/or Calibration Database when executing a calibration products
  production.}

  \item{Compare the implementation of the Template Generation payloads to the
  inputs required by the Alert Production payload as defined in
  \citeds{LDM-151}.}

  \item{Ensure that all data products required by the L1 system are correctly
  produced and persisted appropriately to the LSST Data Backbone when
  executing a template generation production.}

  \item{Demonstrate that QC metrics are properly calculated and transmitted
  during the execution all L2 production types.}

  \item{Demonstrate that post-processing QC analysis of data products can be
  used to identify and report on failures or anomalies in the processing.}

\end{itemize}

\subsection{Tasks and criteria}
\label{sec:tasks}

The following are the major items under test:

\begin{itemize}

  \item{The science payload capable of generating all LSST annual data
  products;}

  \item{Calibration products payloads, run at a variety of cadences, to
  generate calibration products required in the generation of LSST nightly and
  annual data products;}

  \item{The template generation payload capable of generating deep templates
  required for difference imaging in the context of both nightly and annual
  processing.}

  \item{Services capable of scheduling and managing the execution of all of
  the above payloads, marshalling their results, and making them available to
  other parts of the system for analysis or further distribution.}

\end{itemize}

\subsection{Features to be tested}
\label{sec:feat2test}

\begin{itemize}

  \item{Execution of payloads described in \S\ref{sec:tasks};}
  \item{Persistence of all required data products;}
  \item{Scientific fidelity of those data products: do they satisfy the
  requirements described in \citeds{LSE-61}?}

\end{itemize}

\subsection{Features not to be tested}
\label{sec:featnot2test}

This document does not describe facilities for periodically generating or
collecting key performance metrics (KPMs), except insofar as those KPMs are
incidentally measured as part of executing the documented testcases. The KPMs
and the system being used to track KPMs and to ensure compliance with
documented requirements is described in \citeds{LDM-502}.

\subsection{Pass/fail criteria}
\label{sec:passfail}

The results of all tests will be assessed using the criteria described in
\citeds{LDM-503} \S4.

Note that, when executing pipelines, tasks or individual algorithms, any
unexplained or unexpected errors or warnings appearing in the associated log
or on screen output must be described in the documentation for the system
under test. Any warning or error for which this is not the case must be filed
as a software problem report and filed with the DMCCB.

\subsection{Suspension criteria and resumption requirements}
\label{suspension}

Refer to individual test cases where applicable.

\subsection{Naming convention}

All tests are named according to the pattern \textsc{prod-xx-yy} where:

\begin{description}[font=\normalfont\scshape]

  \item{prod}{The product under test. Relevant entries for this document are:
    \begin{description}[font=\normalfont\scshape,topsep=-1.0ex]
      \item[DRP]{The Data Release Production payload and associated service}
      \item[CPP]{The Calibration Products Production payload and associated services}
      \item[TMP]{The Template Generation payload and associated service}
    \end{description}
  }
  \item[xx]{Test specification number (in increments of 10)}
  \item[yy]{Test case number (in increments of 5)}

\end{description}

\input{specs/specs.tex}
\input{cases/cases.tex}


Here is what I got from Michelle Gower in email: 
I'm assuming based upon some comments you've made that we need to merge "Jim's" and my part of the test into an end-to-end test.      If so, we'll need to get the level of details the same.    I'm assuming that the reader shouldn't have to understand all of the details of each piece of the design.    

Here's the first draft of some of the test specs and test report.

======================================================
Test Spec

Note: Once the files have been ingested into the DBB, full tests of the 
   Data Backbone services can be run for more thorough
   checks.    See Data Backbone Services tests for details.    

Procedure:
* If not already available, set up test machines,
  services, accounts (ITC)
    * DBB gateway client machine that can see files output by L1 archiver
    * DBB gateway server machine that has:
        * File transfer service
        * Delivery area
        * Write access to Data Backbone filesystem(s)
    * Database with apropriate schema
    * Authentication to file transfer service and database using LSST AA(?) service.

* Install and configure software
    * DBB Gateway client machine - dbb\_gwclient and prereqs
    * DBB Gateway server machine - dbb\_gateway and prereqs

* If not already running, start test services
    * (L1 test stand services)
    * DBB Gateway
    * Monitoring

* The L1 test stand is triggered to save 1 or more raw fits files.

* After the test L1 archiver has output files,
  the DBB gateway client will stage the files plus appropriate
  information to the test DBB gateway.

* The test DBB Gateway service will automatically
  ingest the staged file into the test DBB which
  includes saving appropriate location, physical, science,
  and provenance information to the database.   Spot check data 
  in the database and the DBB filesystem or run DBB tests.

* The test DBB services will automatically ensure that
  copies of the files go to the right DBB locations as well
  as in disaster recovery.    Spot check files are in the correct 
  locations or run DBB tests.
 
* Run spot checks to ensure meet accessibility requirements or
 run DBB tests.


======================================================================
Test Report

Test systems used:
* The DBB Gateway client was run on user's workstation with no
direct access to the L1 test stand outputs.
* The DBB Gateway service was run on a temporary virtual machine
running HTTPS/webdav.
* The DBB database was a single-node Oracle system.

Differences in procedure for 503-4b:
* (The test file was manually created by Felipe instead of the L1
test stand.)
* The test file was manually copied to a non-L1 test stand machine
to continue rest of test.
* The test DBB gateway client is manually executed.  Integration
work to automate the process is future milestone.  It is expected
* The test DBB Gateway service was also manually executed.
* Since both ends of the process was run by user, only testing user
authentication (as opposed to service accounts)
* The test DBB Gateway service stores no science information in the
DB.  Temporary code in place waiting for Gen3 Butler.
* There was a single site, single copy with no disaster recovery.
* There was no monitoring.
* Did manual checks inside the DBB.

Results:
Within the restrictions for this milestone, the test was successful.
Following the modified test procedure with the available hardware,
an ATS raw file along with information including user and md5sum were
sent via HTTPs to the delivery area on the DBB gateway machine where
another program was executed to copy the raw file to the appropriate
test DBB directory as well as save information to the DBB database
tables.

\appendix

\section{The Hyper Suprime-Cam ``RC'' datasets}

\subsection{RC1}

The original HSC ``release candidate'' dataset was defined as part of testing
release 3.9.0 of the HSC pipeline (derived from the LSST DM Stack). It
consists of 237 visits to the HSC ultra-deep Cosmos field and 83 visits to the
HSC wide survey. Specifically, this includes the following
visits\footnote{Defined using the standard LSST command-line task syntax}:

\subsubsection{Ultra-deep Cosmos}
\label{sec:hscrc1}

\begin{description}

\item[HSC-G]{\hfill \\ 11690..11712:2\^{}29324\^{}29326\^{}29336\^{}29340\^{}29350\^{}29352}
\item[HSC-R]{\hfill \\ 1202..1220:2\^{}23692\^{}23694\^{}23704\^{}23706\^{}23716\^{}23718}
\item[HSC-I]{\hfill \\ 1228..1232:2\^{}1236..1248:2\^{}19658\^{}19660\^{}19662\^{}19680\^{}19682\^{}19684\^{}\\19694\^{}19696\^{}19698\^{}19708\^{}19710\^{}19712\^{}30482..30504:2}
\item[HSC-Y]{\hfill \\ 274..302:2\^{}306..334:2\^{}342..370:2\^{}1858..1862:2\^{}1868..1882:2\^{}11718..11742:2\^{}22602..\\22608:2\^{}22626..22632:2\^{}22642..22648:2\^{}22658..22664:2}
\item[HSC-Z]{\hfill \\ 1166..1194:2\^{}17900..17908:2\^{}17926..17934:2\^{}17944..17952:2\^{}17962\^{}28354..28402:2}
\item[NB0921]{\hfill \\ 23038..23056:2\^{}23594..23606:2\^{}24298..24310:2\^{}25810..25816:2}

\end{description}

\subsubsection{Wide}

\begin{description}

\item[HSC-G]{\hfill \\ 9852\^{}9856\^{}9860\^{}9864\^{}9868\^{}9870\^{}9888\^{}9890\^{}9898\^{}9900\^{}9904\^{}9906\^{}9912\^{}11568\^{}\\11572\^{}11576\^{}11582\^{}11588\^{}11590\^{}11596\^{}11598}
\item[HSC-R]{\hfill \\ 11442\^{}11446\^{}11450\^{}11470\^{}11476\^{}11478\^{}11506\^{}11508\^{}11532\^{}11534}
\item[HSC-I]{\hfill \\ 7300\^{}7304\^{}7308\^{}7318\^{}7322\^{}7338\^{}7340\^{}7344\^{}7348\^{}7358\^{}7360\^{}7374\^{}7384\^{}7386\^{}\\19468\^{}19470\^{}19482\^{}19484\^{}19486}
\item[HSC-Y]{\hfill \\ 6478\^{}6482\^{}6486\^{}6496\^{}6498\^{}6522\^{}6524\^{}6528\^{}6532\^{}6544\^{}6546\^{}6568\^{}13152\^{}13154}
\item[HSC-Z]{\hfill \\ 9708\^{}9712\^{}9716\^{}9724\^{}9726\^{}9730\^{}9732\^{}9736\^{}9740\^{}9750\^{}9752\^{}9764\^{}9772\^{}9774\^{}\\17738\^{}17740\^{}17750\^{}17752\^{}17754}

\end{description}

\end{document}