extend mrca call to take optional argument of plesiomorphic taxa

[mtholder]

response would be the mrca and the series of ancestors down to the next node that is not an ancestor of any "extra" plesiomorphic taxa

[snacktavish]

Potential spec
Inputs
'include':'[list of ottids to be included]'
'exclude':'[list of ottids to be included]'
'type':'hard', 'soft'

Return
would return list of nodes starting with MRCA of all 'include' taxa, to the node 1 more recent than the MRCA of the 'include' taxa and ANY 'exclude taxa '
if type:soft
note if there are any of the 'exclude' taxa nested within the MRCA of the 'include' taxa (e.g. potentially a character state reversal)
if type:hard
if any 'exclude' taxa are nested within the MRCA of the 'include' taxa, FAIL

[bredelings]

OK, I have a draft implementation on the otcetera phyloref branch.

Currently spec is:
Inputs:
"node_ids": [list of node ids to be included]
"exclude_soft": [list of node ids to be excluded]
"exclude_hard": [list of node ids to be excluded]
Return:
400 if MRCA of node_ids not found (existing behavior)
404 if any hard excluded taxa nested in the include group
404 if ALL soft excluded taxa nested in the include group
200 {"node_ids": [list of all nodes from mrca of include group up to exclude group],
"soft_reversals": [list of soft_excluded taxa nested in include group, if not empty] }

Questions: How to handle broken taxa in the exclude groups? (currently, fail)
Questions: Should we add all normal mrca response fields and put everything inside an "mrca" key? (currently, no)

I think I prefer requiring "include" (as @snacktavish suggested above) instead of "node_ids" if an "exclude" field is supplied. I kind of like the ability have both a soft and a hard exclude group, though I don't really have a specific use case.

[jimallman]

400 if MRCA of node_ids not found (existing behavior)

Why wouldn't this be 404 (resource not found) as in the other cases? From my reading, status code 400 means that the request is malformed and cannot be interpreted, but that doesn't seem to be the case here.

Should we add all normal mrca response fields and put everything inside an "mrca" key? (currently, no)

If I understand this correctly, it seems like the MRCA of the include group (first of node_ids in response) is still a major object of interest, and so it might make sense to include these properties just for that node. Or perhaps it's yet another calling option whether to include them?

[bredelings]

@jimallman Yes, I think we should return 404 instead of 400 there.

If I understand correctly, for node-based-names, the MRCA of the include group is of primary interest, but for branch-based-names it could be a node that is slightly more ancestral than that, and for character-based-names all nodes in between the two are equally of interest.

We could still return the MRCA info for the MRCA of the include group... but this feels kind of like this is a new call (phylo_ref?) that is similar to MRCA but fits within a different ecosystem.

So, not sure.

[bredelings]

We have extended the opentree web api so that it can answer phyloref-like queries. You should be able to test it out by performing calls against our "devapi" server (see below).

The tree_of_life/mrca call can now be used to look up node-based names, branch-based names, and apomorphy-based names by adding the additional argument "excluded_nodes_ids". If "excluded_node_ids" is present, the call returns a list of nodes between the MRCA of the "node_ids" (the include group) and the MRCA of the "excluded_node_ids" (the exclude group). For example, the call below looks up a name with an include group of {Rattus,Mus} and an exclude group of {Primates}:

$ curl -X POST https://devapi.opentreeoflife.org/v3/tree_of_life/mrca -d '{"node_ids": ["ott271578","ott1068778"], "excluded_node_ids": ["ott913935"]}'
{
"node_ids": [
"mrcaott102ott8118",
"mrcaott102ott283439",
"ott816256",
"mrcaott102ott739",
"mrcaott42ott102",
"mrcaott42ott55942",
"mrcaott42ott45197",
"ott839752",
"ott7067181",
"mrcaott42ott254702",
"mrcaott42ott48903",
"mrcaott42ott38834",
"mrcaott42ott10477",
"ott864593",
"mrcaott42ott29157",
"ott392220",
"mrcaott42ott30082"
],
"synth_id": "opentree12.3"
}

The first entry ("mrcaott102ott8118"), in the node_ids response is the node-based name. The last entry ("mrcaott42ott30082") is the branch-based name. For an apomorphy-based name, the entire set of nodes should be considered, since the character of interest could have changed anywhere along the path.

Normally, if any excluded nodes are nested inside the clade of the include group, an error is returned. Problematic nodes are listed in a JSON attributed called "reversals". However, for an apomorphy-based name, it is possible that the character of interest could revert to its original state. In that case, one might wish to ignore taxa where the character has reverted. This can be accomplished by passing in "soft_exclude": true, which will just remove problematic taxa from the exclude group instead of causing an error. In that case, problematic nodes are listed in a JSON attributed called "soft_reversals" [1].

[1] Maybe they should be listed as "reversals" in both cases.

[gaurav]

Thanks so much for developing this, @bredelings! I spoke to the rest of the Phyloref team today. We're planning to incorporate support for this API call into our Phyloreference JavaScript library (phyloref/phyx.js#45) and to use it to test whether our OWL-based resolution works correctly (phyloref/clade-ontology#72). I'll let you know if we run into any problems in developing this. Thanks so much again!

[gaurav]

@bredelings Is there any way to use this code on an input phylogeny, or will it only work on the synthetic tree?

[bredelings]

I think the short answer is that it will only work on the synthetic tree.

[hlapp]

Hey guys, is this live on the production server meanwhile? If so, it seems the documentation of the endpoint needs updating with the spec here?

[bredelings]

This has been running on production for a while:

$ curl -X POST https://api.opentreeoflife.org/v3/tree_of_life/mrca -d '{"node_ids": ["ott271578","ott1068778"], "excluded_node_ids": ["ott913935"]}'
{
 "node_ids": [
  "mrcaott102ott8118",
  "mrcaott102ott283439",
  "ott816256",
  "mrcaott102ott739",
  "mrcaott42ott102",
  "mrcaott42ott55942",
  "mrcaott42ott45197",
  "ott839752",
  "ott7067181",
  "mrcaott42ott254702",
  "mrcaott42ott48903",
  "mrcaott42ott38834",
  "mrcaott42ott10477",
  "ott864593",
  "mrcaott42ott29157",
  "ott392220",
  "mrcaott42ott30082"
 ],
 "synth_id": "opentree12.3"
}

[hlapp]

But the documentation does not mention the additional capabilities: https://github.com/OpenTreeOfLife/germinator/wiki/Synthetic-tree-API-v3#mrca

[bredelings]

Hi Hilmar,

I e-mailed you documentation for this back on February 26th. Additionally, that same documentation is posted above in this thread, dated Mar 3rd. My impression was that Guarav or somebody would do some kind of testing and let us know if there were any issues.

Do you have any code that queries this interface? If so, I would be happy to update the public documentation.

If not, I would prefer to wait to post documentation publicly until at least some minimal testing has been done. Otherwise, if you find problems, we would have to consider changing a publicly documented API.

-BenRI

[gaurav]

Hi Ben! Sorry, that's on me -- I wrote some tests for our use of this API and found that ~60% of the clade definitions from Regnum resolved to a node on the Open Tree, but didn't get around to completing testing and writing up those results before I got sidetracked by other work. I'll try to get that done by early next week, and then report that back to you so you can the documentation up at your end. Let me know if that sounds good to you!