Permalink
Fetching contributors…
Cannot retrieve contributors at this time
252 lines (204 sloc) 10.7 KB

Intro

This file documents the development processes that have not (yet) been automated and the practices that developers should follow when working on NIFSTD.

All of these processes expect the full NIF-Ontology development environment to have been set up according to development setup.md.

All relative filepaths listed in code sections start from the base folder of this repository ../ from the location of this file.

DO NOT PUSH ONTOLOGY FILES TO MASTER

The uri.neuinfo.org resolver points to the master branch on GitHub (see the resolver config).

Therefore, please do not push to the master branch.

If you are making changes to the ontology that are more than cosmetic please push to the staging branch. That way merging from staging to master can be accompanied by a SciGraph build and pushed out to services so that everything can stay 'mostly' in sync.

Please submit a pull request (even if it is from another branch in the SciCrunch org repo) so that we can run all the requisite checks to make sure that the ontology is consistent.

Please do push documentation to master

catalog-v001.xml.example should also always be updated on master

Processes

Development

Prior to commit

ttlfmt $(git status -s | grep ttl | grep M | cut -d' ' -f3)
# OR if you are working on files that need to be normalized
qnamefix $(git status -s | grep ttl | grep M | cut -d' ' -f3)

Adding a new external import

All external imports should be imported into NIFSTD via a bridge file. This file will hold any local additions that we want to make.

  1. Create a new bridge file by copying an existing bridge file and modifying as needed to import the external file from its canonical iri, and to import filename-dead.ttl.
  2. Add an entry in catalog-extras for the new import.
  3. Create ttl/generated/filename-dead.ttl by running necromancy http://myurl.org/filename.owl.
  4. Whenever there is a new release repeat step 3.

Changing an ontology file

  1. Edit your file and save.
  2. If using protege git diff to make sure that any new triples have landed in the correct file.
  3. Execute prior to commit processes.

Adding a new ontology file

touch filename.ttl
add filename.ttl
`make_catalog`  # NOTE this is broken at the moment
  1. Add an entry in catalog-v001.xml by hand. Note that you will need to make sure that catalog-v001.xml is copied from the example order to make use of the new entry.

Before you run make_catalog make sure that the import chain is local (unpatched) otherwise it will take an extremely long time to fetch files.

Load remote imports from local copies

Sometimes we don't want to have to retrieve remote copies of files every time we start protege, or we need to run tests for patches that are applied to external imports when loading the ontology into SciGraph.

cd ttl/
cat <(head -n -1 catalog-v001.xml) ../catalog-extras <(tail -n 1 catalog-v001.xml) > catalog-new
mv catalog-new catalog-v001.xml

Local copies can be obtained using the following.

cd ttl
curl --header 'Accept-Encoding: gzip' --compressed --location --create-dirs \
$(cat ../catalog-extras | cut -d'=' -f4 | cut -d'"' -f2 | sed 's/^/-o /') \
--remote-name-all $(cat ../catalog-extras | cut -d'=' -f3 | cut -d'"' -f2)

Running the reasoner

Due to a bug in protege we need to patch our import chain in order to for the reasoner to run correctly.

patch -p1 -i dev-reasoner.patch
# OR
cd ttl/
patch -p2 -i ../dev-reasoner.patch
# Then
protege  # can be launched as you see fit
  1. In protege menu bar File -> Open and choose ttl/nif.ttl (after first load File -> Open recent is faster).
  2. Make sure reasoner is set to ELK (menu bar Reasoner -> ELK 0.4.3)
  3. Ctrl-R

Protege does not have to be run from ttl/ in order to find ttl/catalog-v001.xml, however the file opened must be in ttl/.

Check for incorrect or unimported predicates

  1. Remove /tmp/ttlcmp.patch if it exists
  2. ttlcmp.sh (lives in pyontutils/pyontutils)
  3. Review /tmp/ttlcmp.patch for annotation and object properties.

Integrating a new set of terms

If there is an existing ttl file that has been formatted with ttlfmt as described above then create a new branch, add the file, and submit a pull request for review.

If there is a chance that this file may need to be regenerated from an external source please add the external source and the generation code via the process described in pyontutils/development/README.md. This does not need to be done if ttl file will become the source of truth once integrated into NIFSTD.

Any set of terms that requires more than direct transformation to ttl should also follow the process described in pyontutils/development/README.md.

External source synchronization

Full documentation of defaults and command line arguments can be found by running --help for any of the commands listed in this section.

All of these will soon be set up to run on cron jobs and automatically submit pull requests to the NIF-Ontology repository.

All of these scripts expect pyontutils and the NIF-Ontology repos to be in the same directory ${HOME}/git/pyontutils ${HOME}/git/NIF-Ontology. I am in the process of making this configurable (for some scripts is already is).

The following sections require you to have already run pipenv install --skip-lock and pipenv shell. For registry-sync you may want to run pipenv install --skip-lock --dev in which case see the development installation notes for potential pitfalls.

Start with

cd ~/git/pyontutils &&  # adjust accordingly
pipenv shell

SciCrunch Registry

BRANCH="registry-$(date +%s)"
registry-sync &&
cd ${PATH_TO_ONTOLOGY} &&
git pull &&  # alternately git clone and set pushurl -> script-user or something
git checkout -B ${BRANCH} &&
git push --set-upstream origin ${BRANCH} &&
echo success
# TODO auto github api pull

parcellation

TODO

slimgen

TODO

Release processes

The exact details of the release process are still being worked out so this section may be out of date. Check the underlying documentation and the python scripts/commits for the latest information.

Run sanity checks on staging

Besides the checks listed as part of the development process we need to integrate automated tests, probably using ROBOT. See also https://github.com/SciCrunch/NIF-Ontology/issues/60.

versionInfo

Only bump the versionInfo string on nif.ttl for major releases (e.g. one where nif.ttl is pushed to bioportal). Our versionIRI implementation on top of the unix epoch provides the logical end of granular versioning so we will use that for all practical versioning needs. It seems unlikely that we will bump to a 4.0 release any time in the near future so simply increase the count on the version by one for each release.

versionIRI

Run ontutils version-iri ttl/nif.ttl from the root of the ontology git directory. Then run ontutils iri-commit NIF-Ontology -l ${git-base} and copy and paste the git commit command and execute it. This can now be run on any branch and will resolve correctly when merged to master.

TODO: Should we add a process for adding a versionIRI to individual files whenever there is a substantial non-cosmetic change? It is already possible to use the epoch based versionIRIs to refer to files even if we do not include the versionIRI in the file explicitly.

Merge to master

If the changes are minor don't bother with a pull request, run git checkout master && git merge staging and then push. If there are more major changes, such as changes to modelling of a domain or deprecation of ontology files, submit a pull request, even if you are the only person who will participate in it. This can provide a nice summary for someone who needs to review the history of changes but shouldn't have to dig through the full commit log.

Build graph for SciGraph release

See point 1 in pyontutils/scigraph/README.md#etc-for-centos-7 The build process for the graph is quite customizable. Run ontload scigraph --help if you need additional guidance.

Deploy to SciGraph

This process is documented at pyontutils/scigraph/README.md.

Restart dependent services

  1. Restart the ontree process on aux-resolver. This is needed to flush the cache of the import chain.

Building documentation

Resolver

See pyontutils/resolver/aux-resolver.conf and pyontutils/resolver/nif-ont-resolver.conf.

Setup

See pyontutils/README.md#requirements for details on pandoc and orgstrap. For the best results use >=app-text/emacs-26.1 and >=app-text/pandoc-2.2.3.2. For centos 7 (i.e. the current aux-resolver sever) this means

  1. download, extract, build, and install emacs
  2. download, extract, and install the latest version of pandoc.
  3. git clone https://github.com/tgbugs/orgstrap.git and run orgstrap/orgstrap

Build

On aux-resolver SCICRUNCH_API_KEY=$(cat api-key) pyontutils/pyontutils/docs.py

Deploy

On aux-resolver cp -a pyontutils/pyontutils/doc_build/docs /var/www/ontology-docs

External records

There are a couple of places where there are external records of the ontology that a maintainer or curator needs to keep up to date.

  1. https://scicrunch.org/resolver/SCR_005414 (NIF Ontology)
  2. https://scicrunch.org/resolver/SCR_005402 (NeuroLex)
  3. https://scicrunch.org/resolver/SCR_016178 (InterLex)
  4. https://bioportal.bioontology.org/ontologies/NIFSTD
  5. https://bioportal.bioontology.org/ontologies/NIFSUBCELL
  6. https://fairsharing.org/bsg-s002584 (NeuroLex)
  7. https://fairsharing.org/bsg-s002628 (NIF Ontology)
  8. https://fairsharing.org/bsg-s002835 (subcellular)