Migrate remaining obo format ontologies from dependency on central OBO build

[cmungall]

Original ticket #290

Documentation: http://obofoundry.org/faq/what-is-the-build-field.html

Also: http://www.obofoundry.org/principles/fp-002-format.html Note: some groups publish an .obo version, and the OBO Foundry pipeline takes care of making the valid .owl file. See the FAQ for details. You may also submit the ontology for review as OBO, see ‘criteria for review’ below.

This central job is no longer running. This means that the OWL for some ontologies is lagging behind what the provider intends with their provided obo format version.

We could restart the build, but we think it may be better to get groups set up with providing the OWL themselves. Now with github actions and ODK it should be easier to do this. There is still the concern that many of these groups (e.g model organism communities) may have no technical resources, but we would potentially do this just with PRs in their ontologies

• update http://obofoundry.org/faq/what-is-the-build-field.html to say this no longer runs
• find list of ontologies that rely on this
• work with each such that they are providing owl themselves
• clarify fp-002- such that it is clear each group is responsible for providing their own OWL

[matentzn]

UPDATED 8 July 2023:

Quick analysis of the main purl (owl) redirects, if this helps:

Out of 186 active ontologies

• 137 raw.github to master/main
• 13 raw.github branch/tag (not master) - see below
• 6 github raw - alternative syntax
• 13 Old Berkley Server - see below <- THESE NEED FIXING!
• 18 Github releases (latest)
• 9 other

Purls to raw.github branch/tag:

https://raw.githubusercontent.com/BFO-ontology/BFO/v2019-08-26/bfo_classes_only.owl -- bfo
https://raw.githubusercontent.com/biobanking/biobanking/v2023-04-05/obib.owl -- obib
https://raw.githubusercontent.com/infectious-disease-ontology/infectious-disease-ontology/2017-11-03/ido.owl -- ido
https://raw.githubusercontent.com/information-artifact-ontology/IAO/v2022-11-07/iao.owl -- iao
https://raw.githubusercontent.com/ISA-tools/stato/dev/releases/latest_release/stato.owl -- stato
https://raw.githubusercontent.com/monarch-initiative/GENO-ontology/develop/geno.owl -- geno
https://raw.githubusercontent.com/obi-ontology/obi/v2023-03-10/obi.owl -- obi
https://raw.githubusercontent.com/OHMI-ontology/OHMI/v2019-09-17/ohmi.owl -- ohmi
https://raw.githubusercontent.com/OmniSearch/ncro/release-2015-12-10/build/ncro-all-in-one.owl -- ncro
https://raw.githubusercontent.com/OPL-ontology/OPL/v2023-06-23/opl.owl -- opl
https://raw.githubusercontent.com/safisher/ornaseq/2019-07-08/ornaseq.owl -- ornaseq
https://raw.githubusercontent.com/SED-ML/KiSAO/deploy/kisao.owl -- kisao
https://raw.githubusercontent.com/VEuPathDB-ontology/VEuPathDB-ontology/2023-05-30/eupath.owl -- eupath

Alternative github raw notation

https://github.com/allysonlister/swo/raw/master/swo.owl -- swo
https://github.com/AnimalGenome/vertebrate-trait-ontology/raw/master/current_release/vt.owl -- vt
https://github.com/CTDbase/exposure-ontology/raw/master/exo.owl -- exo
https://github.com/flora-phenotype-ontology/flopoontology/raw/master/ontology/flopo.owl -- flopo
https://github.com/obophenotype/ascomycete-phenotype-ontology/raw/master/apo.owl -- apo
https://github.com/terms4fairskills/FAIRterminology/raw/master/t4fs.owl -- t4fs

berkeleybop.org server:

Remaining

• http://ontologies.berkeleybop.org/ddanat.owl -- ddanat
• http://ontologies.berkeleybop.org/hom.owl -- home
• http://ontologies.berkeleybop.org/ma.owl -- ma
• http://ontologies.berkeleybop.org/spd.owl -- spd
• http://ontologies.berkeleybop.org/zfs.owl -- zfs

Migrated (August 2023):

• http://ontologies.berkeleybop.org/hsapdv.owl -- hsapdv
• http://ontologies.berkeleybop.org/mmo.owl -- mmo
• http://ontologies.berkeleybop.org/mmusdv.owl -- mmusdv
• http://ontologies.berkeleybop.org/olatdv.owl -- olatdv
• http://ontologies.berkeleybop.org/pdumdv.owl -- pdumdv
• http://ontologies.berkeleybop.org/pw.owl -- pw
• http://ontologies.berkeleybop.org/rs.owl -- rs
• http://ontologies.berkeleybop.org/xco.owl -- xco

Rest

https://gitlab.com/fortunalab/ontoavida/-/raw/master/ontoavida.owl -- ontoavida
https://gitub.u-bordeaux.fr/erias/fideo/-/raw/master/fideo.owl -- fideo
https://proconsortium.org/download/current/pro_reasoned.owl -- pr
http://www.informatics.jax.org/downloads/reports/mp.owl -- mp
https://bitbucket.org/hegroup/ogg/raw/master/ogg-merged.owl -- ogg
https://download.rgd.mcw.edu/ontology/clinical_measurement/clinical_measurement.owl -- cmo
http://current.geneontology.org/ontology/go.owl -- go
http://ftp.ebi.ac.uk/pub/databases/chebi/ontology/chebi.owl -- chebi
http://vertebrate-taxonomy-ontology.s3-website-us-west-2.amazonaws.com/releases/vto.owl -- vto

[matentzn]

Code snippet to get the redirects:

import requests
import yaml
import urllib.request
with urllib.request.urlopen("http://obofoundry.org/registry/ontologies.yml") as data:
    obo_data = yaml.safe_load(data)
    
for o in obo_data['ontologies']:
    oid = o['id']
    if o['activity_status']=="active":
            purl = f"http://purl.obolibrary.org/obo/{oid}.owl"
            r = requests.get(purl, allow_redirects=False)
            url = r.headers['Location']
            print(f"{url}||{oid}")

[nlharris]

This came up during today's OFOC call. Looks like it's mostly done?

[matentzn]

Not by a longshot, but @balhoff s idea with #1577 will help speed this up.

[matentzn]

Update 2 years later: 13 ontologies left to redirect. Action item for the next person coming across this issue:

• Contact all 13 ontologies to update their PURL config (tell them how)

[matentzn]

@fbastian considers deprecating olatdv and pdumdv

EDIT: these were set to inactive.

[matentzn]

Today, I contacted

Petra Fey <pfey northwestern.edu>,
bgee@sib.swiss,
Terry Hayamizu <Terry.Hayamizu jax.org>,
"Smith, Jennifer" <jrsmith mcw.edu>,
gthayman mcw.edu,
sjwang mcw.edu,
ramirez macn.gov.ar,
Yvonne Bradford <ybradford zfin.org>

Via email:

Dear ontology maintainers (ddanat, hom, ma, mmo, pw, rs, spd, xco, zfs)!

We have a long-standing issue of ontologies with stale OWL products: 

https://github.com/OBOFoundry/OBOFoundry.github.io/issues/1446

This means that for quite a few years, your ontologies have been stale (never updated) in OBOs primary dependency system, which relies on this PURL: http://purl.obolibrary.org/obo/ID.owl, e.g. http://purl.obolibrary.org/obo/ddant.owl. 

Would you be able to either email me the location of your correct OWL files, or update the purl config yourself, e.g. https://github.com/OBOFoundry/purl.obolibrary.org/blob/master/config/zfa.yml?

Feel free to reach out to me if you have problems or questions, we can provide people to support you.

All best,
Nico (OBO Operations Committee)

[nlharris]

Is this still in progress?

[matentzn]

Latest rerun:

• 140 raw.github
• 19 GitHub releases
• 6 github.com/../raw

13 custom deployment:

purl	space	id
http://current.geneontology.org/ontology/go.owl		go
http://ftp.ebi.ac.uk/pub/databases/chebi/ontology/chebi.owl		chebi
http://vertebrate-taxonomy-ontology.s3-website-us-west-2.amazonaws.com/releases/vto.owl		vto
http://www.informatics.jax.org/downloads/reports/mp.owl		mp
https://bitbucket.org/hegroup/ogg/raw/master/ogg-merged.owl		ogg
https://download.rgd.mcw.edu/ontology/clinical_measurement/clinical_measurement.owl		cmo
https://download.rgd.mcw.edu/ontology/experimental_condition/experimental_condition.owl		xco
https://download.rgd.mcw.edu/ontology/measurement_method/measurement_method.owl		mmo
https://download.rgd.mcw.edu/ontology/pathway/pathway.owl		pw
https://download.rgd.mcw.edu/ontology/rat_strain/rat_strain.owl		rs
https://gitlab.com/fortunalab/ontoavida/-/raw/master/ontoavida.owl		ontoavida
https://gitub.u-bordeaux.fr/erias/fideo/-/raw/master/fideo.owl		fideo
https://proconsortium.org/download/current/pro_reasoned.owl		pr

Remaining onces broken:

purl	space	id
http://ontologies.berkeleybop.org/hom.owl		hom
http://ontologies.berkeleybop.org/ma.owl		ma
http://ontologies.berkeleybop.org/zfs.owl		zfs

@nlharris thanks for the ping! We are slowly but firmly getting there. 3 to go!

Tracking:

• Source for MA ontology? obophenotype/mouse-anatomy-ontology#124 (MA)
• Clarify difference between two versions of ZFS obophenotype/developmental-stage-ontologies#81 (ZFS)