Project: Pilot Large Data Support Service

[cmbz]

Overview

Pilot Harvard Dataverse large data support services using NESE tape resources for several datasets from Harvard affiliates.

Tasks

• Identify prospective data collections
• Confirm pilot participants
• Coordinate with pilot collection owners to estimate collection size and curation needs
• Create Harvard Dataverse collections
• Coordinate with NESE support staff for tape provisioning
• Coordinate with data collection owners, NESE support, and Dataverse team to upload data
• Coordinate with IQSS finance re. costs
• Assess pilot and propose workflow improvements

Process Development and Management

• Develop an intake process including basic and extended consultation process
• Develop and launch RT queue, including service attributes (will map to service offerings)
• Develop large data collection intake form @sbarbosadataverse
• Define what is included in the Large Data Technical Support service component
• Define what is included in the Large Data Administration service component
• Define what is included in the Large Data Monitoring service component
• End-user instructions for NESE data access, add to Dataverse documentation in Guide and HDV support website
• Document large data curation services process (see hdv-curation issues on large data)

Team

• @sbarbosadataverse
• @cmbz
• @qqmyers
• @landreev

Pilot Participants

The Oregon-Massachusetts Mammography Database (OMAMA-DB)

See also: https://fly.cs.umb.edu/omama/ and https://github.com/IQSS/dataverse-HDV-Curation/issues/443

• Size: 8T
• Contact: Daniel Haehn, SEAS & UMass Boston

GEOS-Chem 1 and 10 year Benchmark data

See also:
https://help.hmdc.harvard.edu/Ticket/Display.html?id=361614
https://github.com/IQSS/dataverse-HDV-Curation/issues/456

• Datasets
  • http://ftp.as.harvard.edu/gcgrid/geos-chem/1yr_benchmarks/
  • http://ftp.as.harvard.edu/gcgrid/geos-chem/10yr_benchmark
  • Size: about 1.6 TB:
    • 484G 10yr_benchmarks/
    • 484G total
    • 1.1T 1yr_benchmarks/
    • 1.1T total
  • Note: It would increase as they add new benchmark output. typically release model versions once per quarter. Prior to each release they would generate 1 or more benchmark runs for GCClassic and GCHP. The 10 year benchmarks are generated once per major version release which are less frequent.

DrivAerNet: A Parametric Car Dataset for Data-driven Aerodynamic Design and Graph-Based Drag Prediction

See also: https://github.com/IQSS/dataverse-HDV-Curation/issues/444

• moving their ‘streamlined’ dataset (<1TB) to the DeCoDE lab’s dataverse
• <1TBish
• File sizes: Largest: 1.6G and smallest: 400bytes,
• Total number of files: not sure, user quoted 10^6, but was fuzzy on that.
• File types: .tar.gz, txt, vtk, stl, .sh, data will be packed in .zip format
• By mid to end of May: the full 16TB.

Under discussion for support

-https://help.hmdc.harvard.edu/Ticket/Display.html?id=362556
-https://help.hmdc.harvard.edu/Ticket/Display.html?id=358471

Related RT Tickets

• #356558: Harvard Dataverse support request: Hosting of breast cancer database
  
• https://help.hmdc.harvard.edu/Ticket/Display.html?id=361614

Issues

• IQSS/10450 - Fixes for Globus download (transfer out) and related info messages dataverse#10451
• Prod. globus infrastructure, next phase: Work out and test the logistics of Globus downloads from NESE dataverse.harvard.edu#261
• Prod. globus infrastructure, next phase: Create documentation for data uploaders dataverse.harvard.edu#260

Related

• Service: Harvard Dataverse Repository Large Data Services #268
• GREI 5: HDV Task - Large Data Support #176
• Project: GEOS-Chem Proof-of-Concept #154
• Project: MOC Proof-of-Concept #105
• Start uploading production data to tape storage at NESE dataverse.harvard.edu#254

Resources

• Google Drive Folder

[cmbz]

Status: March 2024

• Our pilot for computing on large data was completed. See follow up: Epic: Operationalize Large Data and Compute Infrastructure #184 and continuing work here: Start uploading production data to tape storage at NESE dataverse.harvard.edu#254

Closed

• Project: MOC Proof-of-Concept #105

[cmbz]

Status: April 2024

Improvements made to Globus: "The PR fixes the issue with multifile Globus transfers out/downloads not working for draft datasets. It also improves handling of cases where ineligible files are selected for download or Globus transfer by only showing the download/transfer mechanisms that will work (on some files) given the files in the dataset and by improving the UI messages to indicate that files may not be eligible either because the user doesn't have permission (restricted or embargoed), or because the files can't be downloaded/transferred (i.e. files not in a Globus store when the user tries a Globus transfer or files in a Globus store that doesn't support normal downloads when the user selects download.)"

• IQSS/10450 - Fixes for Globus download (transfer out) and related info messages dataverse#10451

[sbarbosadataverse]

Status: May 2024

• Added new Process Development and Management section to this issue
• Tested Globus/NESE download for OMAMA dataset, download successful but found that guestbook is not collecting downloads, could be due to the dataset being in "draft"
• Leonid created a demo dataset for testing Globus access for non Harvard affiliates: https://demo.dataverse.org/dataset.xhtml?persistentId=doi:10.70122/FK2/URDDBC
• Ceilyn revised documentation for large data services, budget planner, and internal HDV service components
• New section added to record groups interested, but not yet confirmed for the pilot

[cmbz]

Status: June 2024

• RT for large data support is created and populated (still require attributes we want to track)
• Started budget for Jax labs https://help.hmdc.harvard.edu/Ticket/Display.html?id=363736
• Started Globus instructions for users for OMAMA dataset

[sbarbosadataverse]

Status: July 2024

• Draft intake form started:

• Instructions to download data from NESE complete

• OMAMA collection updated with NESE download instructions and checklist for curation completed (minus Invoice): https://github.com/IQSS/dataverse-HDV-Curation/issues/443

• Estimate for https://github.com/IQSS/dataverse-HDV-Curation/issues/444 completed and shared with Author