From 2d9cd6d8c9cdbfb312892922a7785869a9f82877 Mon Sep 17 00:00:00 2001 From: William O'Mullane Date: Fri, 5 Aug 2022 17:39:55 -0700 Subject: [PATCH 1/5] new changes --- acronyms.tex | 2 ++ proposal.tex | 25 +++++++++++++++++++++---- resources.tex | 11 +++++++++-- 3 files changed, 32 insertions(+), 6 deletions(-) diff --git a/acronyms.tex b/acronyms.tex index 9f452b9..b86b2a9 100644 --- a/acronyms.tex +++ b/acronyms.tex @@ -13,6 +13,7 @@ DMTN & DM Technical Note \\\hline DP0 & Data Preview 0 \\\hline DRP & Data Release Production \\\hline +FITS & Flexible Image Transport System \\\hline FY24 & Financial Year 24 \\\hline IDAC & Independent Data Access Center \\\hline IVOA & International Virtual-Observatory Alliance \\\hline @@ -29,5 +30,6 @@ RTN & Rubin Technical Note \\\hline SDSS & Sloan Digital Sky Survey \\\hline SLAC & SLAC National Accelerator Laboratory \\\hline +TACC & Texas Advanced Computing Center \\\hline USDF & United States Data Facility \\\hline \end{longtable} diff --git a/proposal.tex b/proposal.tex index 5e1f03e..5863e29 100644 --- a/proposal.tex +++ b/proposal.tex @@ -14,15 +14,26 @@ \subsection{Science Platform}\label{sec:rsp} \citeds{DMTN-202} points 4 (\secref{sec:requirements}) on catalogs we can assume means a sort of Dask/Spark access to the Parquet files. The baseline answer for catalog queries was Qserv and but we so not have the "next to the database processing". The DAC team feel the parquet files provide a good mechanism for this and there is some effort in FY24 to implement it.{\textbf We should consider carefully what the minimum system we need here is to cover our requirements.} -We will not at this point promise extensive Dask/Spark like services but should work on that in the background with LINCC. I think we all agree this will be scientifically useful, but we need to finish construction as a priority. +We will not at this point promise extensive Dask/Spark like services but should work on that in the background with LINCC. +We all agree this will be scientifically useful, but we need to finish construction as a priority and this is not a requirement. We will work with LINCC on it certainly and something will be available but we are not promising this and we are not accepting requirements on it. -\textbf{Do we agree to work with LINCC on DASK/Spark ? and promise the minimum batch system and next to the database processing as the construction deliverable} +The standard/default allocation for any data rights user will be RSP access. +Users wilt bulk needs will use BPS (\secref{sec:bpsbatch}) -\subsection{Butler Gen3} + +\subsection{Butler Gen3} \label{sec:gen3} Whether Batch or RSP a butler is available with provenance information which should cover DMS-REQ-0106 and DMS-REQ-0121. +If users do not use the Butler they will not have provenance. \subsection{BPS Batch }\label{sec:bpsbatch} + +Our baseline is user batch based on Gen3 Task framework. +Users needing batch will need to go via the Resource Allocation Committee (\secref{sec:allocation}) and +will be given SLAC accounts to run user Batch jobs at SLAC. +Those who do not wish to use the Rubin pipelines could be facilitated see \secref{sec:othercode}. + + We set DP0.2 as an initial test of seeing how we could work with PanDA \citeds{RTN-013} for DRP. Potentially it could also be used for Prompt Processing and user batch processing. In fact we use the BPS front end to submit jobs and middleware which to keep BPS working with at least on other back end as well as PanDA. @@ -37,6 +48,12 @@ \subsection{BPS Batch }\label{sec:bpsbatch} This would also cover DMS-REQ-0119, DMS-REQ-0125, DMS-REQ-0128, DMS-REQ-0123, DMS-REQ-0127. +\subsubsection{Non Rubin code} \label{sec:othercode} +Users may wish to run arbitrary code on Rubin images. +It should be trivial to provide a butler recipe to get an image or set of images to the users space and allow them to run any code which takes FITS images. Done in a container this could be done for a large volume of images - it would have to get past the resource allocation committee of course. +Run in this mode no provenance would be kept. (See \secref{sec:gen3}) + + \subsection{Arbitration or Resource Allocation Committee}\label{sec:arbitration} As \secref{sec:resources} points out we will need an arbitration committee of some sort. It seems this should also favor individuals from under represented and under resourced groups though that is being proposed here for the first time. @@ -63,5 +80,5 @@ \subsection{Other catalogs} LINCC may again help here by coordinating IDACs to provide neighbor tables/services for other catalogs. One notion would be to have a the Object catalog at IDACs matched to their local holdings. We could consider migrating some of those neighbor catalogs back to USDF Qserv. +Gaia will be provided as part of the processing - there is no requirement listing catalogs so from a verification perspective we can put this on IDACs -\textbf{How do we answer this fairly open requirement ? Do we say we deliver Gaia matched for construction and will work on an IDAC/LINCC service ?} diff --git a/resources.tex b/resources.tex index 5db2fe4..adfaa5a 100644 --- a/resources.tex +++ b/resources.tex @@ -14,6 +14,8 @@ \subsection{Compute} \label{sec:compute} User batch would therefore have about 1K cores depending on load. This implies the need for a controlled sharing mechanism (\secref{sec:quotas}). +It would also be excellent to pursue a research project with an NSF facility (e.g. TACC) into what could be done with Rubin data at such a facility in terms of arbitrary compute. TACC have expressed interest in this. + \subsection{Storage} @@ -38,15 +40,20 @@ \subsection{Quotas and allocation} \label{sec:quotas} This however needs more work. -%{\bf In operations should we use all Google Cores for interactive batch Dask/Spark ? Send all PanDA to USDF ? But then we do not have 10\% there since we spent on Google.} - PROBALBY NOT FEASABLE +In operations we propose to use all Google Cores for interactive batch Dask/Spark. +We will send all batch jobs to USDF. We have some capacity at USDF for this and potentially some spare cycles in the shared data facility. \citeds{DMTN-135} table 40 allocates 2PB in year 1 growing to 9PB in year 10 for science user home space. This is still driven by the original requirement to support 7500 users. Table 40 assumes 5K users in year 1 yielding a limit of 40GB per user. Of course usage will not be even so we could set the limit higher like 60GB and monitor what happens. On SDSS we kept the quota low and accepted justified requests to get more - most people kept within the quota by keeping their user space. Its not obvious what a good number is here but perhaps we should start the RSP users on 10GB quota. +KT points out on NCSA "9\% (81 of 892) of NCSA home directories had more than 10 GB on 2022-02-07." +This would indicate starting allocation on RSP of 10GB may work for 90\% of users - we can then increase on request for some users. + +Power users who would be some of that other 10\% may be more likely to seek batch production at SLAC in any case. + -{\bf Should we start user space quota at 10GB ? is there a reason for more ? (or less?)} It is not clear we have the technology in place to enforce disk quotas yet. From 008ef58aa887078342af9d8af69e1ed2c1ed1a7d Mon Sep 17 00:00:00 2001 From: William O'Mullane Date: Sat, 13 Aug 2022 15:47:41 -0700 Subject: [PATCH 2/5] post meeting changes --- .gitignore | 1 + acronyms.tex | 2 ++ proposal.tex | 6 +++--- resources.tex | 5 ++--- 4 files changed, 8 insertions(+), 6 deletions(-) diff --git a/.gitignore b/.gitignore index d173e69..fc20236 100644 --- a/.gitignore +++ b/.gitignore @@ -17,6 +17,7 @@ one.* *.fot *.cb *.cb2 +*.xdv ## Intermediate documents: diff --git a/acronyms.tex b/acronyms.tex index b86b2a9..7c00395 100644 --- a/acronyms.tex +++ b/acronyms.tex @@ -15,12 +15,14 @@ DRP & Data Release Production \\\hline FITS & Flexible Image Transport System \\\hline FY24 & Financial Year 24 \\\hline +GB & Gigabyte \\\hline IDAC & Independent Data Access Center \\\hline IVOA & International Virtual-Observatory Alliance \\\hline LINCC & LSST Interdisciplinary Network for Collaboration and Computing \\\hline LPM & LSST Project Management (Document Handle) \\\hline LSE & LSST Systems Engineering (Document Handle) \\\hline LSST & Legacy Survey of Space and Time (formerly Large Synoptic Survey Telescope) \\\hline +NCSA & National Center for Supercomputing Applications \\\hline NERSC & National Energy Research Scientific Computing Center \\\hline NSF & National Science Foundation \\\hline PanDA & Production ANd Distributed Analysis system \\\hline diff --git a/proposal.tex b/proposal.tex index 5863e29..0a1b8de 100644 --- a/proposal.tex +++ b/proposal.tex @@ -12,13 +12,13 @@ \subsection{Science Platform}\label{sec:rsp} The RSP is now well established and is following the vision \citeds{LSE-319}. Data Preview 0 has introduced tn early version of this to the community. -\citeds{DMTN-202} points 4 (\secref{sec:requirements}) on catalogs we can assume means a sort of Dask/Spark access to the Parquet files. The baseline answer for catalog queries was Qserv and but we so not have the "next to the database processing". The DAC team feel the parquet files provide a good mechanism for this and there is some effort in FY24 to implement it.{\textbf We should consider carefully what the minimum system we need here is to cover our requirements.} +\citeds{DMTN-202} points 4 (\secref{sec:requirements}) on catalogs we can assume means a sort of Dask/Spark access to the Parquet files. The baseline answer for catalog queries was Qserv and but we so not have the "next to the database processing". The DAC team feel the parquet files provide a good mechanism for this and there is some effort in FY24 to implement it. We should consider carefully what the minimum system we need here is to cover our requirements. We will not at this point promise extensive Dask/Spark like services but should work on that in the background with LINCC. We all agree this will be scientifically useful, but we need to finish construction as a priority and this is not a requirement. We will work with LINCC on it certainly and something will be available but we are not promising this and we are not accepting requirements on it. The standard/default allocation for any data rights user will be RSP access. -Users wilt bulk needs will use BPS (\secref{sec:bpsbatch}) +Users wilt bulk needs will use BPS (\secref{sec:bpsbatch}) at a Data Facility. \subsection{Butler Gen3} \label{sec:gen3} @@ -29,7 +29,7 @@ \subsection{Butler Gen3} \label{sec:gen3} \subsection{BPS Batch }\label{sec:bpsbatch} Our baseline is user batch based on Gen3 Task framework. -Users needing batch will need to go via the Resource Allocation Committee (\secref{sec:allocation}) and +Users needing batch will need to go via the Resource Allocation Committee (\secref{sec:arbitration}) and will be given SLAC accounts to run user Batch jobs at SLAC. Those who do not wish to use the Rubin pipelines could be facilitated see \secref{sec:othercode}. diff --git a/resources.tex b/resources.tex index adfaa5a..b9e67c1 100644 --- a/resources.tex +++ b/resources.tex @@ -66,9 +66,8 @@ \subsubsection{Unused home space} Space on cloud means cost. If a user is not using their space for long period it should be migrated to cheaper storage. Along period could be a year but six months seems long enough. -Again we have no mechanism for this - Richard even suggested taking it to USDF but I wonder if cold storage on google is better. +Again we have no mechanism for this - Richard even suggested taking it to USDF but cold storage on google might work. -{\bf Should we have user home migration to get un touched home spaces moved off cloud?} \subsection{Other sites}\label{sec:othersites} @@ -76,7 +75,7 @@ \subsection{Other sites}\label{sec:othersites} Some science collaborations, most notably DESC, will have their own resources at some center (NERSC for DESC). In the DESC case there are good links between NERSC and SLAC making this easy to manage. -No NSF facility is currently planned to hold the data. +No NSF facility is currently planned to hold the data. TACC are interested to experiment with us. These sites are true batch sites with Slurm workload management. Users would submit jobs with BPS allowing it to figure out if the Parsl, PanDA for job submission on the back end. From b45e62638fe7b4d103db03aaa46d932edfc18fd6 Mon Sep 17 00:00:00 2001 From: William O'Mullane Date: Mon, 29 Aug 2022 21:26:25 +0100 Subject: [PATCH 3/5] Tim --- proposal.tex | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/proposal.tex b/proposal.tex index 0a1b8de..9001fb1 100644 --- a/proposal.tex +++ b/proposal.tex @@ -50,8 +50,13 @@ \subsection{BPS Batch }\label{sec:bpsbatch} \subsubsection{Non Rubin code} \label{sec:othercode} Users may wish to run arbitrary code on Rubin images. -It should be trivial to provide a butler recipe to get an image or set of images to the users space and allow them to run any code which takes FITS images. Done in a container this could be done for a large volume of images - it would have to get past the resource allocation committee of course. -Run in this mode no provenance would be kept. (See \secref{sec:gen3}) +It should be possible to provide a butler recipe to get an image or set of images to the users space and allow them to run any code which takes FITS images. +Done in a container this could be done for a large volume of images - it would have to get past the resource allocation committee of course. +This would require custom work by the user and would not be a general facility. +Run in this mode little provenance would be kept. (See \secref{sec:gen3}) + +Some provenance could be garnered by ingesting the generated files which could at least give the execution graph of their creation. +If users wish only to write files they would have to go in their user space or potentially a VOSpace end point - the user would need to organize and track such files themselves. \subsection{Arbitration or Resource Allocation Committee}\label{sec:arbitration} From 973c39c0d9865819d9287b17562b592d32e72ca5 Mon Sep 17 00:00:00 2001 From: William O'Mullane Date: Tue, 13 Sep 2022 12:10:19 +0100 Subject: [PATCH 4/5] GPDF comments --- acronyms.tex | 1 + proposal.tex | 13 +++++++++---- requirements.tex | 2 +- resources.tex | 4 +++- 4 files changed, 14 insertions(+), 6 deletions(-) diff --git a/acronyms.tex b/acronyms.tex index 7c00395..f7a33ac 100644 --- a/acronyms.tex +++ b/acronyms.tex @@ -27,6 +27,7 @@ NSF & National Science Foundation \\\hline PanDA & Production ANd Distributed Analysis system \\\hline Parsl & Parallel Scripting Library \url{http://parsl-project.org/} \\\hline +RAC & Resource Allocation Committee \\\hline RDO & Rubin Directors Office \\\hline RSP & Rubin Science Platform \\\hline RTN & Rubin Technical Note \\\hline diff --git a/proposal.tex b/proposal.tex index 9001fb1..f5d8129 100644 --- a/proposal.tex +++ b/proposal.tex @@ -10,15 +10,19 @@ \section{Proposal} \label{sec:proposal} \subsection{Science Platform}\label{sec:rsp} The RSP is now well established and is following the vision \citeds{LSE-319}. -Data Preview 0 has introduced tn early version of this to the community. +Data Preview 0 has introduced an early version of this to the community. -\citeds{DMTN-202} points 4 (\secref{sec:requirements}) on catalogs we can assume means a sort of Dask/Spark access to the Parquet files. The baseline answer for catalog queries was Qserv and but we so not have the "next to the database processing". The DAC team feel the parquet files provide a good mechanism for this and there is some effort in FY24 to implement it. We should consider carefully what the minimum system we need here is to cover our requirements. +\citeds{DMTN-202} point 4 (\secref{sec:requirements}) in respect of catalogs we now propose to be satisfied by some form of Dask/Spark access to the catalogs as spatially sharded Parquet files. +The previous baseline design for this was a layer on top of Qserv, but in practice we do not have this ``next to the database processing'' implemented, and the RSP and Data Access teams feel that, with the advance of data-science software, a Parquet-based solution is now a good solution. +There is some effort available in FY24 to prototype and implement this. +We shall consider carefully what the minimum system we need here is to cover our requirements. -We will not at this point promise extensive Dask/Spark like services but should work on that in the background with LINCC. +We will not at this point promise extensive Dask/Spark like services. +The minimum needed to meet requirements will be done on construction and we shall work further on it in the background with LINCC. We all agree this will be scientifically useful, but we need to finish construction as a priority and this is not a requirement. We will work with LINCC on it certainly and something will be available but we are not promising this and we are not accepting requirements on it. The standard/default allocation for any data rights user will be RSP access. -Users wilt bulk needs will use BPS (\secref{sec:bpsbatch}) at a Data Facility. +Users with bulk needs will use BPS (\secref{sec:bpsbatch}) at a Data Facility. \subsection{Butler Gen3} \label{sec:gen3} @@ -62,6 +66,7 @@ \subsubsection{Non Rubin code} \label{sec:othercode} \subsection{Arbitration or Resource Allocation Committee}\label{sec:arbitration} As \secref{sec:resources} points out we will need an arbitration committee of some sort. It seems this should also favor individuals from under represented and under resourced groups though that is being proposed here for the first time. +We note also proposals committing to releasing Level 3 data products that are made available to all data rights holders will also be allowed to be given a preference by the RAC. The committee is called out in the operations plan \citedsp{RDO-018} Section 12.2 but we have as yet not created a charge for it. This committee would at least be responsible for creating, implementing and updating policies on at least: diff --git a/requirements.tex b/requirements.tex index e849d32..dfbae93 100644 --- a/requirements.tex +++ b/requirements.tex @@ -4,7 +4,7 @@ \section{DM requirements on user batch} \label{sec:requirements} has a few requirements which require some sort of user driven processing. Many of these pertain to {\em user generated data products}\footnote{Previously known as Level 3 data products.}. In the era of pre-operations many of these user generated products will be created and may exist at IDACs. -The a broad summary of requirements pertaining to user batch and more broadly to the topic of user generated products and services is covered on confluence\footnote{Should this be in a DMTN or put here ?} \url{https://confluence.lsstcorp.org/display/DM/Level+3+Definition+and+Traceability}{Level 3 Definition and Traceability}. +The a broad summary of requirements pertaining to user batch and more broadly to the topic of user generated products and services is covered in \href{https://confluence.lsstcorp.org/display/DM/Level+3+Definition+and+Traceability}{Level 3 Definition and Traceability}i on confluence. The requirements most pertinent to user batch from \citeds{LSE-61} are : diff --git a/resources.tex b/resources.tex index b9e67c1..f064489 100644 --- a/resources.tex +++ b/resources.tex @@ -40,7 +40,9 @@ \subsection{Quotas and allocation} \label{sec:quotas} This however needs more work. -In operations we propose to use all Google Cores for interactive batch Dask/Spark. +In operations we propose catalog processing with Dask/Spark goes to Google; + any processing (images or, less often, catalogs) done with BPS goes to USDF batch. + We will send all batch jobs to USDF. We have some capacity at USDF for this and potentially some spare cycles in the shared data facility. \citeds{DMTN-135} table 40 allocates 2PB in year 1 growing to 9PB in year 10 for science user home space. From c072f2c771a0389fccacbac84915eec0f580bf4a Mon Sep 17 00:00:00 2001 From: William O'Mullane Date: Tue, 13 Sep 2022 12:34:01 +0100 Subject: [PATCH 5/5] DPDF PDF changes --- proposal.tex | 4 ++-- resources.tex | 6 ++++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/proposal.tex b/proposal.tex index f5d8129..b73edf5 100644 --- a/proposal.tex +++ b/proposal.tex @@ -1,7 +1,7 @@ \section{Proposal} \label{sec:proposal} Let us assume the resources available are fairly fixed in line with \secref{sec:resources}. -Hence we will need an arbitration committee \secref{sec:arbitration} +Hence we will need a resource allocation committee \secref{sec:arbitration} to arbitrate requests. We promised some form of user batch which it seems likely will be needed at least initially as many users understand batch processing of images say. By batch here we should be clear we mean a BPS command line interface to something like PanDA. \secref{sec:bpsbatch} @@ -63,7 +63,7 @@ \subsubsection{Non Rubin code} \label{sec:othercode} If users wish only to write files they would have to go in their user space or potentially a VOSpace end point - the user would need to organize and track such files themselves. -\subsection{Arbitration or Resource Allocation Committee}\label{sec:arbitration} +\subsection{Resource Allocation Committee}\label{sec:arbitration} As \secref{sec:resources} points out we will need an arbitration committee of some sort. It seems this should also favor individuals from under represented and under resourced groups though that is being proposed here for the first time. We note also proposals committing to releasing Level 3 data products that are made available to all data rights holders will also be allowed to be given a preference by the RAC. diff --git a/resources.tex b/resources.tex index f064489..be8e733 100644 --- a/resources.tex +++ b/resources.tex @@ -48,7 +48,7 @@ \subsection{Quotas and allocation} \label{sec:quotas} \citeds{DMTN-135} table 40 allocates 2PB in year 1 growing to 9PB in year 10 for science user home space. This is still driven by the original requirement to support 7500 users. Table 40 assumes 5K users in year 1 yielding a limit of 40GB per user. Of course usage will not be even so we could set the limit higher like 60GB -and monitor what happens. On SDSS we kept the quota low and accepted justified requests to get more - most people kept within the quota by keeping their user space. Its not obvious what a good number is here but perhaps we should start the RSP users on 10GB quota. +and monitor what happens. On SDSS and BaBar the quota was kept low and justified requests to get more were accepted - most people kept within the quota by keeping their user space. Its not obvious what a good number is here but perhaps we should start the RSP users on 10GB quota. KT points out on NCSA "9\% (81 of 892) of NCSA home directories had more than 10 GB on 2022-02-07." This would indicate starting allocation on RSP of 10GB may work for 90\% of users - we can then increase on request for some users. @@ -61,8 +61,10 @@ \subsection{Quotas and allocation} \label{sec:quotas} \subsubsection{Peak times} There are going to be known events such as a Data Release which we know will put pressure on the system. -For these event we would want to scale back to basic quotas like 0.5 core per user and disabling DASK or even batch execution. +For these event we would want to scale back to basic quotas like 0.5 core per user and potentially disabling DASK. We should also experiment with paying for the surge and see if subsequent lulls even out the bill. +The latter approach would allow us to maintain a good service level and would be a more equitable approach since it would +not penalize the users who have no other form of access. \subsubsection{Unused home space} Space on cloud means cost.