update SemMedDB APIs

[andrewsu]

splitting out SemMedDB from the broader issue of updating APIs on pending #25

In addition to updating to the latest data files, we will remove the logic to convert to biolink model from the parser, leaving that to be done in the smartAPI mapping.

[r76941156]

@andrewsu The parsers are almost finished. Need some suggestions for below item.

Last time we added a new item for reverse direction and this is not supported by original SEMMED predication (e.g., treated_by for treats). I assume this task can't be done without using inverse predication from biolink model. So, I am thinking to use below approach:

Current:
"USES" (semmed): {
"forward": "has_input", (biolink)
"reverse": "is_input_of" (biolink)
}

Future:
"USES" (semmed): {
"forward": "uses", (semmed)
"reverse": "is_input_of" (biolink)
}

Forward direction uses original predication but reserver direction uses prediction from biolink. Or please let me know if you have better suggestions. Thanks.

[andrewsu]

Hmm, good question. I'm not in love with the result (because uses is not obviously the reverse of is_input_of). But having said that, I can't think of a better solution. Going to loop in @colleenXu in case they see anything we should consider.

[newgene]

Per our discussion, the current plan is to create a single SemMedDB API with each JSON object as a triple.

Like this existing API: https://biothings.ncats.io/text_mining_targeted_association/query, where each object looks like this:

{
    "_id": "CHEBI:100147-PR:000023452-ay-9l3RTMPjsZ_Jwei4NxBVYBjo",
    "_score": 1,
    "association": {
        "edge_label": "negatively_regulates_entity_to_entity",
        "evidence": [
            {
                "object_spans": "start: 41, end: 45",
                "provided_by": "TMProvider",
                "publications": "PMID:25080814",
                "relation_spans": "",
                "score": "0.99955386",
                "sentence": "The nalidixic acid significantly reduced ompF expression, while kanamycin, for which porins are considered to be an alternative transport route, only slightly reduced the ompF level.",
                "subject_spans": "start: 4, end: 18"
            }
        ],
        "evidence_count": "1",
        "relation": "RO:0002212"
    },
    "object": {
        "PR": "PR:000023452",
        "id": "PR:000023452",
        "type": "GeneOrGeneProduct"
    },
    "subject": {
        "CHEBI": "CHEBI:100147",
        "id": "CHEBI:100147",
        "type": "ChemicalSubstance"
    }
}

Basically this is planned right now:

• Parsing the latest SemMedDB data files into the triple objects (@r76941156)
• We will reuse or modify from the existing GraphQueryHandler implemented in Handler for BioThings API providing graph type data #20 to provide the query feature on either subject, object or association (predicate). Should support batch query too.
• Update SmartAPI metadata to annotate the supported edge queries
• Test in BTE to integrate this API

While we are implementing the GraphQueryHandler, we can start to verify SmartAPI/BTE can support the features we expect.

[n19810426]

@andrewsu @newgene @colleenXu
We have around 25 million edges after processing all SEMMED predications since many PMIDs share the same triple objects, so have to combine them into one record.

I prepared one example file for your reference (Should I follow the format in the word doc or the above format?). Please let me know if you want to adjust the format, especially for below 2 items:

1. One CUI may associate many semantic types:
   "PharmacologicSubstance|AminoAcid,Peptide,orProtein|ImmunologicFactor": [
   {
   "id": "C0003241",
   "name": "Antibodies",
   "pmid": [
   "24639825"
   ]
   }

2. SEMMED DB has 'negation' relations (e.g., NEG_INHIBITS), however the biolink model does not mention/process them particularly. But, I still processed them with PMIDs to keep the original negation information.

"affects": {
"MolecularFunction": [
{
"id": "C0001038",
"name": "Acetylation",
"pmid": [
"6336235",
"2680595",
"25051713",
"24115404",
"20798904",
"2663856",
"18349284",
"10030449(negation)",
"9164413",
"2681143"
]
}

In addition, since we process the SEMMED DB independently, some data checks will be handled by the translator. For example,

1. Check if CUIs and predicates are supported by Biolink model through using Biolink-model YAML and SemanticTypes Mapping
2. Check if domain and range are within the Biolink scopes based the doc (suggested by Stuppie)

I am still doing some tests with DrugMechDB and issue #227 and may adjust something later. Will keep you posted.

[andrewsu]

A couple quick thoughts on your questions above.

1. I see that specific CUIs can be associated with many semantic types, but each record in the predications table only has a single type for each CUI, right? If so, just use the stated type. (If that's wrong, please show an example record.)

2. I think you should treat the negations as completely separate predicates, i.e., "NEG_STIMULATES" is completely different than "STIMULATES".

And a few additional notes:

• Agreed that the biolink mappings will happen in BTE.
• I will let @colleenXu comment on the structure and how it relates to the SmartAPI mappings
• It looks like you are removing whitespaces from the semantic types. I would also prefer removing punctuation and enforcing camel case (e.g., BodyPartOrganOrOrganComponent instead of BodyPart,Organ,orOrganComponent)
• What version of semmed are you using? I'm comparing to the latest file semmedVER43_2021_R_PREDICATION.csv.gz that I downloaded from https://lhncbc.nlm.nih.gov/ii/tools/SemRep_SemMedDB_SKR/SemMedDB_download.html and I see quite different results for C1412045

[r76941156]

A couple quick thoughts on your questions above.

1. I see that specific CUIs can be associated with many semantic types, but each record in the predications table only has a single type for each CUI, right? If so, just use the stated type. (If that's wrong, please show an example record.)

The current structure is one (subject cui) to many (object cuis). Thus, the semantic type of subject maybe different while dealing with different object CUIs (example).

2. I think you should treat the negations as completely separate predicates, i.e., "NEG_STIMULATES" is completely different than "STIMULATES".

No problem. BTE should also take care this while doing the mapping process.

And a few additional notes:

• Agreed that the biolink mappings will happen in BTE.
• I will let @colleenXu comment on the structure and how it relates to the SmartAPI mappings
• It looks like you are removing whitespaces from the semantic types. I would also prefer removing punctuation and enforcing camel case (e.g., BodyPartOrganOrOrganComponent instead of BodyPart,Organ,orOrganComponent)

I refered the doc used last time for this example. I will change this per suggested request.

• What version of semmed are you using? I'm comparing to the latest file semmedVER43_2021_R_PREDICATION.csv.gz that I downloaded from https://lhncbc.nlm.nih.gov/ii/tools/SemRep_SemMedDB_SKR/SemMedDB_download.html and I see quite different results for C1412045

I used the latest version (07/2021). One column (either subject cui or object cui) may contain multiple ids (example). They are not cuis so have to be processed first (e.g., Entrez ID: 1 <-> C1412045). So, you will see different results.

[colleenXu]

My understanding was that we were moving to an association-based format (see the comment above from Chunlei #30 (comment)), not a entity-based format (which is where there's 1 subject to many objects). In the association-based format, there's just "triples": one set of subject-predicate-object. Is that not correct?

It's interesting that one CUI may be assigned to a different semantic type, depending on which association it's in. The association-based format should work well with that though.

[colleenXu]

I'm also not sure what from Sander means...:

"since many PMIDs share the same triple objects, so have to combine them into one record"

It seems that with association-based modeling, each triple is an object that holds publication info under the association field. I think it's alright if multiple objects have the same PMID under their association field...

---

Also, I believe there's no need to create "reverse" triples, so I'm not sure what this comment from Sander means...

I am still doing some tests with DrugMechDB and issue #227 and may adjust something later. Will keep you posted.

---

Finally, what does it mean when the subject or object CUI field has multiple IDs? This is from Sander's most recent comment:

One column (either subject cui or object cui) may contain multiple ids (example). They are not cuis so have to be processed first (e.g., Entrez ID: 1 <-> C1412045). So, you will see different results.

Are they equivalent IDs for the same entity or IDs for different entities that are supposed to be considered as some kind of "group" for this association? That should help determine if they should be included as equivalent IDs in one association/triple, or whether this has to be split into multiple triples...

[andrewsu]

@colleenXu yes, thank you, you've pointed out some key points. I think the fundamental issue is that each document should be a single row in the predications document, not an entity-centric collection of multiple rows. So @r76941156 your example file needs to be pretty fundamentally reorganized. See the example (from a different resource) that Chunlei posted in this comment #30 (comment)

I think there is a secondary question of what it means when the IDs have | in them (like in @r76941156's example here). Sander, can you find some info in the semmeddb documentation?

[r76941156]

@andrewsu @colleenXu

Please see the new example file and let me know if you want to change anything. All information are from SEMMED DB. The current '_id' field is unique (predication_id) so it should fine for our use. For some CUI id columns that have '|' between ids, I did not find any relevant information in SEMMED's documents. Based on comments from Stuppie and Mike, these non-CUI ids should be gene ids. You are free to browse relevant articles based on their PMIDs to see if you want a better presentation format. They used to process them into CUIs and used them in the '_id' columns.

[andrewsu]

Super, thanks... So just to confirm, when | appears in the SUBJECT_CUI or OBJECT_CUI field, you remove whatever comes after the |? If yes, that looks like appropriate behavior to me.

I propose the following tweaks to the structure:

• moving association.edge_label to predicate (to match the wording used in the semmed download table, and to have the predicate and pmid on the same level; I'm also fine with having them both under association)
• change subject._id and object.id to subject.umls to object.umls
• add in the novelty fields for subject and object (I think we will add a filter to our BTE queries based on these fields)

The modified structure would look something like this:

    {
        "_id": "PREDICATION_ID:10592604",
        "predicate": "process_of",
        "subject": {
            "umls": "C0003725",
            "name": "Arboviruses",
            "semantic_type": "virs",
            "novelty": "1"
        },
        "object": {
            "umls": "C0999630",
            "name": "Lepus capensis",
            "semantic_type": "mamm",
            "novelty": "1"
        },
        "pmid": "16530475"
    },

Let's give @newgene and @colleenXu one day to chime in with any additional feedback. If none, go ahead and push forward with preparing that modified file please...

[colleenXu]

@andrewsu @r76941156

(FYI: I'm going to use my understanding of the columns of the predication table, but perhaps there's already been discussions of what columns to keep and what to exclude...My comments below are not intended to be a final word or override previous decisions...)

---

What about:

1. If we use the structure of previous "association-based modeling", I suggest putting the info about the predication into a "predication" section - which functions like the association section. What do others think?
2. "Translating" the semantic type codes into human-readable camel case. I believe that's been mentioned above.
3. Is there always 1 PMID per predication? Or is it possible to find two predications that only differ by the PMID? (I ask in case we want to merge them into 1 record...)
4. Is it possible to pull from the sentence table? If not, it's alright but the Unsecret Team's users have liked this feature...

Like the below example (taken from the example Sander's provided)

    {
        "_id": "PREDICATION_ID:10602185",
        "predication": {
            "predication_id": 10602185,   // maybe this isn't needed if it's in _id...
            "predicate": "converts_to",    // semmed's predicate
            "pmid": "16530493",
            "sentence": "hi GSAGs are cool and are digested into Disaccharides that the body uses",      // not real, the idea is to use the SENTENCE_ID field to pull out the sentence...
        },
        "subject": {
            "umls": "C0017973",
            "name": "Glycosaminoglycans",
            "semantic_type": "SomeChemicalGrouping",    // translating the semantic type but I don't know what the original term 'bacs' stands for
            "novelty": 0    // I'm assuming we'd give the original value, not our interpretation of what this value means (which is that this is a more general term)
        },
        "object": {
            "umls": "C0012611",
            "name": "Disaccharides",
            "semantic_type": "Carbohydrate",    // translating the semantic type from "carb"
            "novelty": 0
        }
    }

[r76941156]

@colleenXu
You can find two predications that only differ by the PMID.

@andrewsu
I parsed every id in these CUI fields since we are not sure whether they are the same or highly relevant after conversions. If not the same, there will be new records for them.

[andrewsu]

After a quick check in with @newgene, let's go with the following:

• each row in the predication file corresponds to one object -- no combining when only PMIDs differ
• predicate and pmid at top level
• no sentence info for now
• keep the original case and spacing for predicate and semantic type
• keep semantic type abbreviation and name separately (mapping file can be found at https://metamap.nlm.nih.gov/Docs/SemanticTypes_2018AB.txt)
• pmid as an integer, not a string

Example record below:

    {
        "_id": "PREDICATION_ID:10602185",
        "predicate": "PROCESS_OF",    
        "pmid": 16530493,
        "subject": {
            "umls": "C0017973",
            "name": "Glycosaminoglycans",
            "semantic_type_abbreviation": "bacs",
            "semantic_type_name": "Biologically Active Substance",    
            "novelty": 0    
        },
        "object": {
            "umls": "C0012611",
            "name": "Disaccharides",
            "semantic_type_abbreviation": "carb",
            "semantic_type_name": "Carbohydrate",    
            "novelty": 0
        }
    }

And a bit of a more detailed note on the cases where the SUBJECT_CUI or OBJECT_CUI is not a simple UMLS CUI.

• In cases where a UMLS CUI is followed by one or more numeric IDs (presumed to be NCBI Gene IDs) e.g., C0056207|3075, discard the numeric IDs and process as usual
• In cases where the CUI only consists of one or more pipe-separated numeric IDs, create separate documents for each numeric id using the key ncbigene. For example, this record

10682982,1517,16530800,PART_OF,3346|7435|10614,HTN1|VIS1|HEXIM1,gngm,1,C1481993,Haloviruses,virs,1,\N,\N,\N

would get converted to three separate records that look something like this:

    {
        "_id": "PREDICATION_ID:10682982",
        "predicate": "PART_OF",    
        "pmid": 16530800,
        "subject": {
            "ncbigene": "3346",
            "name": "HTN1",
            "semantic_type_abbreviation": "gngm",
            "semantic_type_name": "Gene or Genome",    
            "novelty": 1    
        },
        "object": {
            "umls": "C1481993",
            "name": "Haloviruses",
            "semantic_type_abbreviation": "virs",
            "semantic_type_name": "Virus",    
            "novelty": 1
        }
    },
    {
        "_id": "PREDICATION_ID:10682982",
        "predicate": "PART_OF",    
        "pmid": 16530800,
        "subject": {
            "ncbigene": "7435",
            "name": "VIS1",
            "semantic_type_abbreviation": "gngm",
            "semantic_type_name": "Gene or Genome",    
            "novelty": 0    
        },
        "object": {
            "umls": "C1481993",
            "name": "Haloviruses",
            "semantic_type_abbreviation": "virs",
            "semantic_type_name": "Virus",    
            "novelty": 1
        }
    },
    {
        "_id": "PREDICATION_ID:10682982",
        "predicate": "PART_OF",    
        "pmid": 16530800,
        "subject": {
            "ncbigene": "10614",
            "name": "HEXIM1",
            "semantic_type_abbreviation": "gngm",
            "semantic_type_name": "Gene or Genome",    
            "novelty": 0    
        },
        "object": {
            "umls": "C1481993",
            "name": "Haloviruses",
            "semantic_type_abbreviation": "virs",
            "semantic_type_name": "Virus",    
            "novelty": 1
        }
    },

Please post if anything isn't clear or if there are other edge cases that need to be considered. Thanks!

[colleenXu]

I'd like clarification on why we wouldn't merge predications that differ only by PMID (and their unique ID). I believe this is the approach taken by the previous SEMMED APIs, and I think it makes sense from a BTE perspective (having 1 edge with multiple supporting publications). (but side note: BTE's response-transform could potentially merge records that only differ by PMID too...)

Is it to keep this data close to its original format? Or perhaps it is related to the "multiple ID" issues described above (where the predication gets split into 3 objects)?

[andrewsu]

The primary motivation is to keep the format/structure as close to the raw data as possible. When we consume it in BTE (or at very least before we send the data back out), we'll be transforming it to "1 edge with multiple supporting publications"...

[r76941156]

@andrewsu @newgene

1. The above example (i.e., PREDICATION_ID:10682982) will produce records with duplicated IDs. I assume this will have issues while indexing the records. Will we use other index approach or ?

2. For some ncbigene cases(e.g., PREDICATION_ID:11064740): It has subject_cui (5515/8634) and object_cui (4954/5290). We will expand one predication record into 4 records (2*2). Is that correct?

3. I think the above SemanticTypes mapping version (2018AB) is the latest one. However, some records from SEMMED used mappings from other versions (e.g., 2013AA) at the time when they created these records. I assume they did not re-parse them when there is a new mapping version. So, these old mappings (e.g., carb) could not be found in the latest version .

For now, I will not add the 'semantic_type_name' for them. Or please let me know your preferred format. Thanks.

[andrewsu]

1. The above example (i.e., PREDICATION_ID:10682982) will produce records with duplicated IDs. I assume this will have issues while indexing the records. Will we use other index approach or ?

Ahh right, great point. the pipe-separated gene IDs was the one thing I didn't discuss with @newgene. I suggest the following then:

• add a numeric suffix when necessary, and drop the curie prefix. So, "_id": "PREDICATION_ID:10682982" would become "_id": "10682982_1", "_id": "10682982_2", etc.
• add a new top-level key for predication_id, which in all cases above would have "predication_id": 10682982

2. For some ncbigene cases(e.g., PREDICATION_ID:11064740): It has subject_cui (5515/8634) and object_cui (4954/5290). We will expand one predication record into 4 records (2*2). Is that correct?

right, that is what I am proposing

3. I think the above SemanticTypes mapping version (2018AB) is the latest one. However, some records from SEMMED used mappings from other versions (e.g., 2013AA) at the time when they created these records. I assume they did not re-parse them when there is a new mapping version. So, these old mappings (e.g., carb) could not be found in the latest version. For now, I will not add the 'semantic_type_name' for them. Or please let me know your preferred format. Thanks.

It looks like the 2018 version only deleted a few semantic types. In that case, let's just use the 2013 version. If there are any semantic types that are even older that don't exist in the 2013 file, then yes, I think not adding the semantic_type_name would be appropriate.

$ diff SemanticTypes_2018AB.txt SemanticTypes_2013AA.txt
23a24
> carb|T118|Carbohydrate
42a44
> eico|T111|Eicosanoid
78a81
> lipd|T119|Lipid
90a94
> nsba|T124|Neuroreactive Substance or Biogenic Amine
93a98
> opco|T115|Organophosphorus Compound
121a127
> strd|T110|Steroid

[newgene]

add a numeric suffix when necessary, and drop the curie prefix. So, "_id": "PREDICATION_ID:10682982" would become "_id": "10682982_1", "_id": "10682982_2", etc.

This sounds good to me. In this Semmed case, _id value is only for internal purpose, not used for the direct queries. So we just need to make them unique for each object. Adding _x is one way, you can also use <predication_id>_<pmid> pattern, if pmid is the field differentiating objects with the same predication_id.

[r76941156]

@andrewsu @newgene

Revised with Andrew's suggestions (less space requried). Please see the example.

The example has 3 cases (normal, without 'semantic_type_name', multiple records for one predication id). Please let me know which one you prefer or you still need some format changes. Thanks.

[andrewsu]

Every row in the csv file should be one document. (CLARIFICATION: When multiple IDs are present in a single row, break that up so that becomes like it was multiple separate rows.) There should be no nesting of multiple rows in a single document (so no introduction of the the semmed key that is present in both of your new examples). Please start with the example I gave in this comment, and then make only the minor modification mentioned in the follow up comment. (I think @newgene's suggestion in his comment is good as a general principle, but in this case I don't believe the <predication_id>_<pmid> pattern will result in a unique _id in this case, because the same <pmid> will be reused for all genes.)

    {
        "_id": "10682982_1",
        "predicate": "PART_OF",    
        "predication_id": 10682982,
        "pmid": 16530800,
        "subject": {
            "ncbigene": "3346",
            "name": "HTN1",
            "semantic_type_abbreviation": "gngm",
            "semantic_type_name": "Gene or Genome",    
            "novelty": 1    
        },
        "object": {
            "umls": "C1481993",
            "name": "Haloviruses",
            "semantic_type_abbreviation": "virs",
            "semantic_type_name": "Virus",    
            "novelty": 1
        }
    },
    {
        "_id": "10682982_2",
        "predicate": "PART_OF",    
        "predication_id": 10682982,
        "pmid": 16530800,
        "subject": {
            "ncbigene": "7435",
            "name": "VIS1",
            "semantic_type_abbreviation": "gngm",
            "semantic_type_name": "Gene or Genome",    
            "novelty": 0    
        },
        "object": {
            "umls": "C1481993",
            "name": "Haloviruses",
            "semantic_type_abbreviation": "virs",
            "semantic_type_name": "Virus",    
            "novelty": 1
        }
    },
    {
        "_id": "10682982_3",
        "predicate": "PART_OF",    
        "predication_id": 10682982,
        "pmid": 16530800,
        "subject": {
            "ncbigene": "10614",
            "name": "HEXIM1",
            "semantic_type_abbreviation": "gngm",
            "semantic_type_name": "Gene or Genome",    
            "novelty": 0    
        },
        "object": {
            "umls": "C1481993",
            "name": "Haloviruses",
            "semantic_type_abbreviation": "virs",
            "semantic_type_name": "Virus",    
            "novelty": 1
        }
    },

[r76941156]

@andrewsu @newgene @colleenXu @erikyao

I have done some tests with limited records on my local machine and su08 and the result looks good to me. Now, the GitHub repo for SEMMED DB is ready to be deployed on pending hub. After deployment, I will do more tests based on the whole dataset and BTE can also test related new features. If anything I can help while implementing new logic with BTE, please let me know. Thanks.

[newgene]

@erikyao Let's use SemMedDB as the API name and association as the doc_type.

[newgene]

Existing SemMedDB APIs are callled semmed*, which eventually will be replaced by this new one.

[erikyao]

Published to https://pending.biothings.io/semmeddb

Please let me know if I can be of any further help.

[newgene]

same as https://biothings.ncats.io/semmeddb

A few example queries:

###
POST  https://biothings.ncats.io/semmeddb/query
Content-Type: application/json

{
    "q": [["C0087111", "C0023570"]],
    "scopes": ["subject.umls", "object.umls"]
}

###
POST  https://biothings.ncats.io/semmeddb/query
Content-Type: application/json

{
    "q": [["TREATS", "Neoplastic Process"]],
    "scopes": ["predicate", "object.semantic_type_name"]
}

where q can have more query items like ["C0087111", "C0023570"] or ["TREATS", "Neoplastic Process"], matching the number and order of the fields in scopes.

[newgene]

Next step is to create the API metadata and test the new features from biothings/call-apis.js#30, @colleenXu? I can review and merge it to main if it does not bring in breaking changes.

[colleenXu]

x-bte annotation has been done for the latest release, and BTE has been switched to use semmeddb instead of the old semmed apis...

[colleenXu]

Note that we're encountering issues with outdated UMLS identifiers in the semmed data...TranslatorSRI/NodeNormalization#119 (comment).

Things to consider:

• having only 1 record for the pipe-delimited stuff - and using the ID that isn't outdated (aka ncbigene)...
• checking if UMLS IDs are outdated or not during ingestion

This causes issues for BTE since there's no cross-mapping / label retrieval during ID resolution...

[colleenXu]

Noting that SEMMEDDB recently released new data files (publications up to feb 2022). In addition with the "outdated identifiers in associations" issue, we may want to look at updating this pending API...