new changes after meeting

.gitignore

@@ -17,6 +17,7 @@ one.*
 *.fot
 *.cb
 *.cb2
+*.xdv
 
 
 ## Intermediate documents:

acronyms.tex

@@ -13,21 +13,26 @@
 DMTN & DM Technical Note \\\hline
 DP0 & Data Preview 0 \\\hline
 DRP & Data Release Production \\\hline
+FITS & Flexible Image Transport System \\\hline
 FY24 & Financial Year 24 \\\hline
+GB & Gigabyte \\\hline
 IDAC & Independent Data Access Center \\\hline
 IVOA & International Virtual-Observatory Alliance \\\hline
 LINCC & LSST Interdisciplinary Network for Collaboration and Computing \\\hline
 LPM & LSST Project Management (Document Handle) \\\hline
 LSE & LSST Systems Engineering (Document Handle) \\\hline
 LSST & Legacy Survey of Space and Time (formerly Large Synoptic Survey Telescope) \\\hline
+NCSA & National Center for Supercomputing Applications \\\hline
 NERSC & National Energy Research Scientific Computing Center \\\hline
 NSF & National Science Foundation \\\hline
 PanDA &  Production ANd Distributed Analysis system \\\hline
 Parsl & Parallel Scripting Library \url{http://parsl-project.org/} \\\hline
+RAC & Resource Allocation Committee \\\hline
 RDO & Rubin Directors Office \\\hline
 RSP & Rubin Science Platform \\\hline
 RTN & Rubin Technical Note \\\hline
 SDSS & Sloan Digital Sky Survey \\\hline
 SLAC & SLAC National Accelerator Laboratory \\\hline
+TACC & Texas Advanced Computing Center \\\hline
 USDF & United States Data Facility \\\hline
 \end{longtable}

proposal.tex

@@ -1,7 +1,7 @@
 \section{Proposal} \label{sec:proposal}
 
 Let us assume the resources available are fairly fixed in line with \secref{sec:resources}.
-Hence we will need an arbitration committee \secref{sec:arbitration}
+Hence we will need a resource allocation  committee \secref{sec:arbitration} to arbitrate requests.
 
 We promised some form of user batch which it seems likely will be needed at least initially as many users understand batch processing of images say. By batch here we should be clear we mean a BPS command line interface to something like PanDA. \secref{sec:bpsbatch}
 
@@ -10,19 +10,34 @@ \section{Proposal} \label{sec:proposal}
 
 \subsection{Science Platform}\label{sec:rsp}
 The RSP is now well established and is following the vision \citeds{LSE-319}.
-Data Preview 0 has introduced tn early version of this to the community.
+Data Preview 0 has introduced an early version of this to the community.
 
-\citeds{DMTN-202} points 4 (\secref{sec:requirements}) on catalogs we can assume means a sort of Dask/Spark access to the Parquet files.  The baseline answer for catalog queries  was Qserv and but we so not have the "next to the database processing". The DAC team feel the parquet files provide a good mechanism for this and there is some effort in FY24 to implement it.{\textbf We should consider carefully what the minimum system we need here is to cover our requirements.}
+\citeds{DMTN-202} point 4 (\secref{sec:requirements}) in respect of catalogs we now propose to be satisfied by some form of Dask/Spark access to the catalogs as spatially sharded Parquet files.
+The previous baseline design for this was a layer on top of Qserv, but in practice we do not have this ``next to the database processing'' implemented, and the RSP and Data Access teams feel that, with the advance of data-science software, a Parquet-based solution is now a good solution.
+There is some effort available in FY24 to prototype and implement this.
+We shall consider carefully what the minimum system we need here is to cover our requirements.
 
-We will not at this point promise extensive Dask/Spark like services but should work on that in the background with LINCC. I think we all agree this will be scientifically useful, but we need to finish construction as a priority.
+We will not at this point promise extensive Dask/Spark like services.
+The minimum needed to meet requirements will be done on construction and we shall  work further  on it in the background with LINCC.
+We all agree this will be scientifically useful, but we need to finish construction as a priority and this is not a requirement. We will work with LINCC on it certainly and something will be available but we are not promising this and we are not accepting requirements on it.
 
-\textbf{Do we agree to work with LINCC on DASK/Spark ? and promise the minimum batch system and next to the database processing  as the construction deliverable}
+The standard/default allocation for any data rights user will be RSP access.
+Users with bulk needs will use BPS (\secref{sec:bpsbatch}) at a Data Facility.
 
-\subsection{Butler Gen3}
+
+\subsection{Butler Gen3} \label{sec:gen3}
 
 Whether Batch or RSP a butler is available with provenance information which should cover DMS-REQ-0106 and DMS-REQ-0121.
+If users do not use the Butler they will not have provenance.
 
 \subsection{BPS Batch }\label{sec:bpsbatch}
+
+Our baseline is user batch based on Gen3 Task framework.
+Users needing batch will need to go via the Resource Allocation Committee (\secref{sec:arbitration}) and
+will be given SLAC accounts to run user Batch jobs at SLAC.
+Those who do not wish to use the Rubin pipelines could be facilitated see \secref{sec:othercode}.
+
+
 We set DP0.2 as an initial test of seeing how we could work with PanDA \citeds{RTN-013}
 for DRP. Potentially it could also be used for Prompt Processing and user batch processing.
 In fact we use the BPS front end to submit jobs and middleware which to keep BPS working with at least on other back end as well as PanDA.
@@ -37,9 +52,21 @@ \subsection{BPS Batch }\label{sec:bpsbatch}
 This would also cover DMS-REQ-0119, DMS-REQ-0125, DMS-REQ-0128, DMS-REQ-0123, DMS-REQ-0127.
 
 
-\subsection{Arbitration or Resource Allocation Committee}\label{sec:arbitration}
+\subsubsection{Non Rubin code} \label{sec:othercode}
+Users may wish to run arbitrary code on Rubin images.
+It should be possible to provide a butler recipe to get an image or set of images to the users space and allow them to run any code which takes FITS images.
+Done in a container this could be done for a large volume of images - it would have to get past  the resource allocation committee of course.
+This would require custom work by the user and would not be a general facility.
+Run in this mode little provenance would be kept.  (See \secref{sec:gen3})
+
+Some provenance could be garnered by ingesting the generated files which could at least give the execution graph of their creation.
+If users wish only to write files they would have to go in their user space or potentially a VOSpace end point - the user would need to organize and track such files themselves.
+
+
+\subsection{Resource Allocation Committee}\label{sec:arbitration}
 As \secref{sec:resources} points out we will need an arbitration committee of some sort.
 It seems this should also favor individuals from under represented  and under resourced groups though that is being proposed here for the first time.
+We note also  proposals committing to releasing Level 3 data products that are made available to all data rights holders will also  be allowed to be given a preference by the RAC.
 The committee is called out in the operations plan \citedsp{RDO-018} Section 12.2 but we have as yet not created a charge for it.
 This committee would at least be responsible for creating, implementing and updating  policies on at least:
 
@@ -63,5 +90,5 @@ \subsection{Other catalogs}
 LINCC may again help here by coordinating IDACs to provide neighbor tables/services for other catalogs.
 One notion would be to have a the Object catalog at IDACs matched to their local holdings.
 We could consider migrating some of those neighbor catalogs back to USDF Qserv.
+Gaia will be provided as part of the processing - there is no requirement listing catalogs so from a verification perspective we can put this on IDACs
 
-\textbf{How do we answer this fairly open requirement ? Do we say we deliver Gaia matched for construction and will work on an IDAC/LINCC service ?}

requirements.tex

@@ -4,7 +4,7 @@ \section{DM requirements on user batch} \label{sec:requirements}
 has  a few requirements which require some sort of user driven processing.
 Many of these pertain to {\em user generated data products}\footnote{Previously known as Level 3 data products.}.
 In the era of pre-operations many of these user generated products will be created and may exist at IDACs.
-The a broad summary of requirements pertaining to user batch and more broadly to the topic of user generated products and services is covered on confluence\footnote{Should this be in a DMTN or put here ?} \url{https://confluence.lsstcorp.org/display/DM/Level+3+Definition+and+Traceability}{Level 3 Definition and Traceability}.
+The a broad summary of requirements pertaining to user batch and more broadly to the topic of user generated products and services is covered in  \href{https://confluence.lsstcorp.org/display/DM/Level+3+Definition+and+Traceability}{Level 3 Definition and Traceability}i on confluence.
 
 The requirements most pertinent to user batch from \citeds{LSE-61} are :
 

resources.tex

@@ -14,6 +14,8 @@ \subsection{Compute} \label{sec:compute}
 User batch would therefore have about 1K cores depending on load.
 This implies the need for a controlled sharing mechanism (\secref{sec:quotas}).
 
+It would also be excellent to pursue a research project with an NSF facility (e.g. TACC) into what could be done with Rubin data at such a facility in terms of arbitrary compute. TACC have expressed interest in this.
+
 
 
 \subsection{Storage}
@@ -38,38 +40,46 @@ \subsection{Quotas and allocation} \label{sec:quotas}
 This however needs more work.
 
 
-%{\bf In operations should we use all Google Cores for interactive batch Dask/Spark ? Send all PanDA to USDF ? But then we do not have 10\% there since we spent on Google.} - PROBALBY NOT FEASABLE
+In operations we propose catalog processing with Dask/Spark goes to Google;
+ any processing (images or, less often, catalogs) done with BPS goes to USDF batch.
+
+We will  send all batch jobs to USDF. We have some capacity at USDF for this and potentially some spare cycles in the shared data facility.
 
 \citeds{DMTN-135} table 40 allocates 2PB in year 1 growing to 9PB in year 10 for science user home space.
 This is still driven by the original requirement to support 7500 users. Table 40 assumes 5K users in year 1
 yielding a limit of 40GB per user. Of course usage will not be even so we could set the limit higher like 60GB
-and monitor what happens. On SDSS we kept the quota low and accepted justified requests to get more - most people kept within the quota by keeping their user space. Its not obvious what a good number is here but perhaps we should start the RSP users on 10GB quota.
+and monitor what happens. On SDSS and BaBar the quota was kept low and justified requests to get more were accepted - most people kept within the quota by keeping their user space. Its not obvious what a good number is here but perhaps we should start the RSP users on 10GB quota.
+
+KT points out on NCSA "9\% (81 of 892) of NCSA home directories had more than 10 GB on 2022-02-07."
+This would indicate  starting allocation on RSP of 10GB may work for 90\% of  users - we can then increase on request for some users.
+
+Power users who would be some of that other 10\% may be more likely to seek batch production at SLAC in any case.
 
 
-{\bf Should we start user space quota at 10GB ?  is there a reason for more ? (or less?)}
 
 It is not clear we have the technology in place to enforce disk quotas yet.
 
 \subsubsection{Peak times}
 There are going to be known events such as a Data Release which we know will put pressure on the system.
-For these event we would want to scale back to basic quotas like 0.5 core per user  and disabling DASK or even batch execution.
+For these event we would want to scale back to basic quotas like 0.5 core per user  and potentially disabling DASK.
 We should also experiment with paying for the surge and see if subsequent lulls even out the bill.
+The latter approach would allow us to maintain a good service level and would be  a more equitable approach since it would
+not penalize the users who have no other form of access.
 
 \subsubsection{Unused home space}
 Space on cloud means cost.
 If a  user is not using their space for long period it should be migrated to cheaper storage.
 Along period could be  a year but six months seems long enough.
-Again we have no mechanism for this - Richard even suggested taking it to USDF but I wonder if cold storage on google is better.
+Again we have no mechanism for this - Richard even suggested taking it to USDF but cold storage on google might work.
 
-{\bf Should we have user home migration to get un touched home spaces moved off cloud?}
 
 
 \subsection{Other sites}\label{sec:othersites}
 SLAC have some potential cores available on site to give more batch possibilities.
 
 Some science collaborations, most notably DESC, will have their own resources at some center (NERSC for DESC).  In the DESC case there are good links between NERSC and SLAC making this easy to manage.
 
-No NSF facility is currently planned to hold the data.
+No NSF facility is currently planned to hold the data. TACC are interested to experiment with us.
 
 These sites are true batch sites with Slurm workload management. Users would submit jobs with BPS
 allowing it to figure out if the Parsl, PanDA for job submission on the back end.