change syn file format

[cbizon]

Changes the structure of the synonyms files to

1. Be in jsonl format
2. include a preferred_name label
3. include biolink types

This new style can be loaded into the prototype name resolver.

While working on this PR, we also fixed a bug in NCBIGene synonym generation.

[gaurav]

Looks good! This code doesn't currently run because synonyms is a set() and not a list() (see inline comment), but that should be a simple fix. I would also recommend changing name to names if it's going to be a set or list of names instead of a single name.

[cbizon]

Can't fully test b/c of BMT bug, but I think that this fixes the problems.

[gaurav]

Thanks, Chris! I'll wait for the BMT bug (biolink/biolink-model-toolkit#111) to be fixed before reviewing this.

[gaurav]

I tested this out on "anatomy" and it works great! Let me know if you'd like to have a look at the generated synonym files. Otherwise, I'm planning to rerun all of Babel early next week and can set up a NameRes loaded from the updated synonym files then.

[gaurav]

One last thing that doesn't make sense: apparently this code (and my code) all worked fine even on bmt v0.9.0 today, not v1.0.4 as fixed by Sierra! I do not understand and do not like pip. I'll upgrade to v1.0.4 and recheck before approving this PR.

[gaurav]

I've confirmed that this works fine on bmt 1.0.4. I think we're good to merge!

[gaurav]

I tried loading this into Solr, but I found something unexpected: note that synonym_factory.get_synonyms(node) doesn't return a list of synonyms, but a list of tuples in the form ('http://www.geneontology.org/formats/oboInOwl#hasExactSynonym', 'Caudal articular process of eighteenth thoracic vertebra'), where the first element is either oboInOwl:hasExactSynonym or the relation included in the synonyms file. When saved into JSON, this is turned into a two dimensional array:

{ "names": [
  ["http://www.geneontology.org/formats/oboInOwl#hasExactSynonym", "Caudal articular process of eighteenth thoracic vertebra"],
  ["http://www.geneontology.org/formats/oboInOwl#hasExactSynonym", "Caudal articular process of eighteenth thoracic vertebra (body structure)]"]
] }

I assume we want to keep this format in Solr. However, when I try loading these JSON files into a Solr instance, it flattens this into a single list of strings:

{"names":[
  "[http://www.geneontology.org/formats/oboInOwl#hasExactSynonym, Caudal articular process of eighteenth thoracic vertebra]",
  "[http://www.geneontology.org/formats/oboInOwl#hasExactSynonym, Caudal articular process of eighteenth thoracic vertebra (body structure)]"]
}

This still works, since you can still search for "Caudal" and get back this result, but all the names also match "hasExactSynonym", which seems non-ideal.

Do we want to keep the synonym types in the synonym output? If not, I could remove that we could simplify this to just a list of names. If we do, I can look into whether we can include more complex data into Solr. @cbizon What do you think?

[cbizon]

Hmm, it's nice to have in the file for backtracking, but it's of little use in the solr database. So I guess I would take it out of the synonyms files. I would say that we could have an intermediate step where we filter them out, so we have one set that we can use for reference and another that is for loading, but the files are already so large that doubling them seems like a bad idea.

So I'm leaning towards just producing the synonyms without the synonym type.

[gaurav]

• There are fewer synonyms being emitted for some terms (e.g. UBERON:0000178 has 13 synonyms in 2022dec2-2 but only five with this code). I wonder if we previously considered labels to also be synonyms but don't do that any longer, or if there's some other general reason for this drop. (Synonyms missing in synonym output #128)
  • Most of those synonyms are associated with UMLS:C0005767, which is a secondary identifier in the UBERON:0000178 clique.
  • Aha, that clique has now been split. Don't know why, but it's not likely to be related to this PR.
• Some leftover UMLS synonyms produced with this code only had a single synonym (e.g. UMLS:C0000715 only has "House, Slaughter") but in 2022dec2-2 these had many more synonyms (e.g. UMLS:C0000715 had ten, including various spellings of "abattoirs" and "slaughterhouses"). This is probably a bug in the UMLS export I wrote. (fixed in 83e83c7)

[gaurav]

I'm pretty sure I've found and fixed all the immediate issues with this PR, so I think it's fine to merge it in. Some additional fixes are in PR #124.