How should we manage the KGOs for the RTW?

[ehogan]

We would like the community's input on how we should manage the Known Good Outputs (KGOs) for the Recipe Test Workflow (RTW; see ESMValTool: Issue: Recipe Test Workflow (RTW) prototype).

• Where should KGOs be stored? (This perhaps relates to the Community: Discussion: Observational Datasets in ESMValTool?)
• Outputs may differ based on the site they were produced at; how should this be managed?

Please add your thoughts and ideas here! 😊

[ehogan]

Here is a summary of my investigation into this issue:

Introduction

There are several aspects to the issue of managing KGOs:

• Storage: The KGOs must be stored somewhere.
• Versioning: Each KGOs must be versioned in some way, so that a known, approved version of each KGO can be used in the Recipe Test Workflow (RTW).
• Access: The KGOs must be accessible to the RTW.
• Updating: There must be a clear process for updating KGOs.

Analysing recipe output

Analysing the output on DKRZ from the recipe testing for ESMValTool v2.9.0 from July 4, 2023:

• Total number of recipes: 153
• Total number of recipe outputs: 87,054
• Total size of recipe outputs: 174 GiB

Breakdown of the total number of recipe outputs

• 21,056 .txt
• 18,853 .xml
• 18,802 .bibtex
• 14,675 .nc
• 9,682 .png
• 2,466 .yml
• 753 .ncl
• 390 .npy
• 153 .html
• 87 .csv
• 84 .asc
• 36 without extension (namelists?)
• 6 .grid
• 4 .pdf
• 2 .mat
• 2 .RDS
• 2 .rsav
• 1 .jpg

Breakdown of the total size of recipe outputs

File format	Total size [KiB]	Total size [GiB]	Average file size	Maximum file size	Notes
.nc	140,416,912	~133.9	~9.3 MiB	~1.6 GiB	37 files > 594 MiB
.txt	12,789,568	~12.2	~607 KiB	~3.0 GiB	18 files >100 MiB
.png	1,541,364	~1.5	~159 KiB	~3.2 MiB

• Remaining ~26 GiB on the other file formats

Requirement questions to consider

• Who will need access to the KGOs?
• What files do we consider to be our KGOs?
  • The image files i.e., plots?
  • The pre-processed netCDF files?
  • The post-processed netCDF files?
  • All of the above?
• What is the maximum size of a given KGO?
• What is the total size of all KGOs?
• How frequently will the KGOs need updating?
• How frequently will the KGOs need to be checked out?
• Can the files be compressed for better performance (size and speed)?

Our requirements

• Those updating the KGOs and those wishing to use KGOs for testing purposes will need access to the KGOs.
• Initially, we consider our KGOs to be the image files. This may be extended to include netCDF files in the future.
• KGOs will need to be updated every time a change is made to the code that changes a KGO i.e., likely multiple times between releases.
• KGOs for a given recipe must be able to be updated independently from other KGOs i.e., it should not be necessary to update the version(s) of all KGOs if only 1 KGO has changed.
• KGOs will need to be checked out once after every time they have been updated, so they can be used by the RTW.

My questions

• The debug page from the v2.9.0 release doesn't provide any information about file sizes; is this something that could be added?
• Are the pre-processed files stored on DKRZ when performing recipe testing?

Storage

Locally

• We store the KGOs locally, either on JASMIN or DKRZ. A detailed and manual process would need to be followed to manage the KGOs, see, for example, the "Versioning" section below.

Git Large File Storage (LFS)

• Git LFS can be used to store files >100MiB.

Git and GitHub

• Git repositories hosted on GitHub can be used to store files <100MiB, see GitHub Docs: About large files on GitHub.
• Git repositories hosted on GitHub should ideally be less than 1 GiB, and less than 5 GiB is strongly recommended, see GitHub Docs: Repository size limits.
• Note: The total size of .png files is currently ~1.5 GiB, where the average file size is ~159 KiB and the maximum file size is ~3.2 MiB. Perhaps this is do-able without Git LFS?

Git LFS and GitHub

• Every account using Git LFS receives 1 GiB of free storage and 1 GiB a month of free bandwidth. If the bandwidth and storage quotas are not enough, you can choose to purchase an additional quota for Git LFS, see GitHub Docs: About storage and bandwidth usage.
  • If we choose this option, KGOs will need to be stored in their own repository within the ESMValGroup organisation, since downloading any file tracked by Git LFS counts towards the repository owner's allotted bandwidth, see GitHub Docs: Tracking storage and bandwidth use. For example, if a KGO was stored in the ESMValTool repository and tracked by Git LFS, any time anyone cloned / pulled ESMValTool, the size of that file would count towards the ESMValTool repository's bandwidth of 1 GiB.
• Git LFS has maximum file size limits when hosting on GitHub, see GitHub Docs: About Git Large File Storage:
  • GitHub Free: 2 GiB.
  • GitHub Enterprise Cloud: 5 GiB.
• Git LFS does not play nicely with Docker, see Add support for Git LFS. docker/hub-feedback#500.

Cloud storage providers

• Amazon S3 (AWS):
  • Amazon S3 has a maximum file size limit of 5 GiB, see Amazon S3: FAQ.
  • Charges apply when storing objects in an S3 bucket, depending on the objects' size, how long you stored the objects during the month, and the storage class, see Amazon S3 pricing: Storage & requests.
  • We could check out up to 100 GiB per month for free, but over that charges apply, see Amazon S3 pricing: Data requests.
• Microsoft Azure Blob Storage:
  • Azure Blob Storage has a block blob maximum file size limit of 4.88 GiB, see Azure documentation: About block blobs.
• Google Cloud Storage (GCP).

Research data repositories

• Note: I need to look at these in more detail 😊
• See e.g., https://www.teamscopeapp.com/blog/6-repositories-to-share-your-research-data:
  • figshare: Free accounts on figshare can upload files of up to 5 GiB and get 20 GiB of free storage.
  • Mendeley Data: Researchers can upload and store their work free of cost on Mendeley Data.
  • Dryad Digital Repository: Charges applied each time a dataset is published.
  • Harvard Dataverse: Harvard Dataverse is free and has a limit of 2.5 GiB per file and 10 GiB per dataset.
  • Open Science Framework: OSF is a free service.
  • Zenodo:
    • Zenodo helps researchers receive credit by making research results citable and through OpenAIRE integrates them into existing reporting lines to funding agencies like the European Commission, see About Zenodo.
    • Zenodo does not impose any requirements on format, size, access restrictions or licence, see About Zenodo.
    • It is possible to upload a dataset containing up to 100 files with a total volume of 50 GiB, see Zenodo documentation: Manage files.
• B2SHARE:
  • B2SHARE is a user-friendly, reliable and trustworthy way for researchers, scientific communities and citizen scientists to store, publish and share research data in a FAIR way, see EUDAT: B2SHARE.
  • B2SHARE Basic is a free-to-use public service that supports the free upload of stable research datasets <20 GiB, which will receive a DOI, see EUDAT: B2SHARE: Brochure.
• ESGF?

DagsHub

• Like GitHub for data science and machine learning, see DagsHub: FAQ.
• Uses DVC (see "Access" section).
• See also DagsHub integration with GitHub.
• Free plan: 100 GiB of DagsHub storage, see DagsHub: pricing.

Versioning

• If the KGOs are stored locally, the checksum of the KGO could be used to version each KGO.
• If the KGOs are stored in a Version Control System (VCS), the Secure Hash Algorithm (SHA) of the commit used to create the KGO could be used to version each KGO.
  • A given KGO may have a different SHA to another KGO, depending on which commit was used to create the KGO.
• If the KGOs are stored in a research data repository, the Digital Object Identifier (DOI) of the KGO would be used to version each KGO.
  • There would need to be one DOI per KGO.

Access

Data Version Control (DVC)

• Open-source, Git-based data science.
• See also DVC: User Guide: Comparison of Related Technologies.
• DVC provides access to external storage locations, see DVC: Remote storage.

Intake

• Intake is a lightweight Python package for finding, investigating, loading and disseminating data.

Pooch

• Pooch makes it easy to download a file.

Updating

Here is a proposed process for updating KGOs:

• Open a new issue to detail which KGO(s) need updating, identifying the change to the code that caused the change to the KGO.
• Create the new KGO(s), and record the new checksum, SHA, or DOI on the issue.
• Ensure the RTW passes using the new KGO(s).
• The new checksum, SHA or DOI of the KGO(s) should be stored in a (YAML?) file within ESMValTool; update the checksum, SHA or DOI in this file (the RTW would use this file to identify which KGO to use).
• Update the KGO(s) in the storage location, ensuring the new KGO(s) reference the new checksum, SHA or DOI appropriately (exact process TBD).

Creating KGOs for the first time

This task must be performed either by the recipe maintainer or someone with the ability to scientifically validate the outputs:

• Open a new issue, one for each recipe that requires KGOs to be produced.
• Run the recipe using a known version of ESMValTool (and ESMValCore).
• Determine which of the text files (.txt) or image files (.png or .jpg) output by the recipe should be used as KGOs.
• Scientifically validate these files.
• Update the issue to formally record:
  • the version of ESMValTool (and ESMValCore) that was used to create the files that were scientifically validated.
  • which files have been scientifically validated.
  • which files should become KGOs.
• Send to the TLT for technical review.

Considerations

• It might be worth considering a phased approached when implementing the solution to the issue of managing KGOs, for example:
  • Phase 1: Single site / local solution.
  • Phase 2: Multi-site / distributed solution.

Conclusions

• Given the limit on the size of repositories hosted on GitHub, only the png files would be able to be hosted on GitHub.
• Something like Zenodo has a better size limit, but the number of files is limited, which would prevent its use.
• It is not obvious to me that there is a single, free solution that will fulfil our requirements.

Does anyone else have any other / additional thoughts?