Re-ID Studies in Napa, verify schema compliance, and ensure pre-requisites for Prod Re-ID

[mbthornton-lbl]

Test Runs on Napa instance and pre-requisites for Metagenomic workflows.
Note:
Context for this test run assumes the the following updates have been applied to the testDB instance:

• Updates to study, biosample, and omics_processing
• Updates for Metaproteomic, Metabolomic and NOM workflows and data

microbiomedata/issues#532

Re-ID Studies:

1. "Stegen": nmdc:sty-11-aygzgv51, formerly gold:Gs0114663
2. "SPRUCE": nmdc:sty-11-33fbta56, formerly gold:Gs0110138
3. "EMP": nmdc:sty-11-547rwq94, formerly gold:Gs0154244
4. "Luquillo": nmdc:sty-11-076c9980, formerly gold:Gs0128850
5. "CrestedButte" nmdc:sty-11-dcqce727, formerly gold:Gs0135149
6. "DeepShale" nmdc:sty-11-8fb6t785, formerly gold:Gs0114675
7. "Populus": nmdc:sty-11-1t150432, formerly gold:Gs0103573

Pre-Requisites - complete ETL on Napa instance and verify:

• Re-ID, Ingest to Napa DB, and Verify Napa compliance for "Luqillo" study nmdc:sty-11-076c9980 #1793
• Re-ID, Ingest to Napa DB, and Verify Napa compliance for "SPRUCE" study nmdc:sty-11-33fbta56 #1789
• Re-ID, Ingest to Napa DB, and Verify Napa compliance for "EMP" study: nmdc:sty-11-547rwq94 #1792
• Re-ID, Ingest to Napa DB, and Verify Napa compliance for "CrestedButte" study nmdc:sty-11-dcqce727 #1794
• Re-ID, Ingest to Napa DB and Verify Napa compliance for "Stegen" study: nmdc:sty-11-aygzgv51 #1814
• Re-ID, Ingest to Napa DB, and Verify Napa compliance for "DeepShale" study: nmdc:sty-11-8fb6t785 #1835
• Re-ID, Ingest to Napa DB, and Verify Napa compliance for "Populus" study nmdc:sty-11-1t150432 #1836
• Extract Re-IDed Napa DB to YAML, Validate and check referential integrity #1865

Pre-requisites - All ETL recipes fully reproducable:

Study, BioSample and Omics:

• Add logic for Napa-compliance updating of Study, BioSamples and OmicsProcessing to re_id_tool.py  nmdc_automation#66
• Handle emsl:output does not match nmdc:dobj failures vs. v10.1 schema #1833
• Handle nmdc:nnn does not match nmdc:dobj errors #1834

Metagenomics:

• Ensure metagenomics re_id_tool.py ready for prod run
• Implement Re-ID function for Metaproteomic Omics Processing and Data Object records nmdc_automation#87
• Metabolomics / NOM Review and test NOM re-IDing code #1928

[mbthornton-lbl]

SPARQL query for Orphaned DataObjects:

PREFIX nmdc: <https://w3id.org/nmdc/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
# Orphan DataObjects - not object of has_input or has_output
select * where { 
    ?do a nmdc:DataObject .
    minus {
        ?o nmdc:has_input ?do .
    }
    minus {
        ?o nmdc:has_output ?do .
    }
} limit 100 

[aclum]

I reopened SPRUCE but this likely pertains to all the studies. We are missing deleting some of the binning data object records. ie
{'description':{$regex:/Gp0208377/}}
from the SPRUCE example data object type are Metagenome Bins,CheckM Statistics or null. The null ones, based on this example, could be captured by a case insensitive search for metabat2 on slot description

[mbthornton-lbl]

@aclum Are we deleting all Binning data objects, or only those with a non-compliant ID?
Should records like this one:

record: nmdc:dobj-11-qm3fbt63 CheckM Statistics CheckM for nmdc:wfmag-11-m0t5hc17.1

be deleted?

NO only non-compliant identifiers

[ssarrafan]

@mbthornton-lbl will be continuing to work on this in the next sprint per Slack message. Moving over.