Rejected evidence - see queries/use-cases.txt

[samuelrosko]

The current model does not include where evidence is rejected, there's also no mp predicate for indicating that

[timclark]

Sam - would you like to have a skype call on this issue?

Briefly, having a predicate for “rejection” is one way to do this - but not how I would recommend approaching the problem.

What I would do is one or both of two things. (1) use a qualifier “REJECTED” or "REJECTED BY KB CURATOR"- qualifiers of all kinds are supported in the model. You can also have an “ACCEPTED” tag (2) if you want to track WHY it is rejected (and/or, why its inverse is ACCEPTED, then assert another claim stating why and citing the evidence you find credible.

The authorship of the accepting or rejecting claims is “KB Curator”, who is privileged in terms of what s/he accepts or rejects being the KB point of view at any given time.

You are essentially raising the question of evidence weighting or credibility in an indirect way. REJECTED is a stronger form of challengedBy and ACCEPTED is a stronger form of supportedBy, with implied authorship “KB Curator”. I would not set up new predicates in that way however.

---

Tim Clark, Ph.D.
Assistant Professor of Neurology, Harvard Medical School
Director of Informatics, MassGeneral Institute for Neurodegenerative Disease
co-Director, Data and Statistics Core, Massachusetts Alzheimer Disease Research Center
website: http://mindinformatics.org
mobile: +1 617-947-7098 fax: +1 617-213-5418

On Jul 1, 2015, at 5:16 PM, Samuel Rosko notifications@github.com wrote:

The current model does not include where evidence is rejected, there's also no mp predicate for indicating that

—
Reply to this email directly or view it on GitHub #15.

[timclark]

HI All,

This is an important topic for the DIKB because, as a policy, we track
all evidence that is rejected. Rejections are based on the explicit
inclusion criteria plus the curator's judgement, and a reason for
rejection is always required. Rejected evidence can apply to a single
claim or a whole "family" of claims that share some commonality. For
example, a poorly done in vitro experiment conducted to assess the
metabolism of Drug X can be rejected for all "Drug X substrate-of Y"
claims.

I could see qualifiers working for this use case with two additional
features:

1. the resource that the qualifer points to should be a simple graph
   that we can use to query out the reason for rejection, who rejected it,
   and when.

2. Rather than the claim itself, the subject of qualifiedBy would need
   to be attached to one of MP data, materials, or method resources in the
   support or refutation chain for a claim. Otherwise, we would be
   incorrectly stating that the curator rejects the claim rather than the
   use of an evidence item as support or refutation for that claim.

To flesh this out a bit more, here is snippet of RDF points to a claim
and provides some data (assuming that having EV_PK_DDI_NR_Data as a
sub-class of MP:Data is correct). I have added in mp:qualifiedBy
predicate pointing to a made up resource that would later be described
elsewhere with the reason for rejection etc. Since this is data, maybe
the rejection was because of bad statistical methods etc.

<rdf:Description 
rdf:about="http://purl.org/net/nlprepository/spl-ddi-annotation-poc#ddi-spl-annotation-data-24">
     <rdf:type 
rdf:resource="http://dbmi-icode-01.dbmi.pitt.edu/dikb-evidence/DIKB_evidence_ontology_v1.3.owl#EV_PK_DDI_NR_Data"/>
     <mp:supports 
rdf:resource="http://purl.org/net/nlprepository/spl-ddi-annotation-poc#ddi-spl-annotation-claim-25"/>
    *<mp: qualifiedBy 
rdf:resource="http://purl.org/net/nlprepository/spl-ddi-annotation-poc#ddi-rejection-1"/>*
<dikbD2R:increases_auc>1.069</dikbD2R:increases_auc>
   </rdf:Description>

However, when I look at
http://www.jbiomedsem.com/content/5/1/28/figure/F5 and at MP n Protege
, I see that the range of mp:qualifiedBy is mp:Sentence. I don't see it
in the ontology, but it doesn't seem that MP:Sentence can be an RDF
resource like described above. If that is true, and the above make
sense, do we need a different property for handling rejected evidence?

ty,
-R

On 07/01/2015 05:32 PM, Tim Clark wrote:

Sam - would you like to have a skype call on this issue?

Briefly, having a predicate for “rejection” is one way to do this -
but not how I would recommend approaching the problem.

What I would do is one or both of two things. (1) use a qualifier
“REJECTED” or "REJECTED BY KB CURATOR"- qualifiers of all kinds are
supported in the model. You can also have an “ACCEPTED” tag (2) if you
want to track WHY it is rejected (and/or, why its inverse is ACCEPTED,
then assert another claim stating why and citing the evidence you find
credible.

The authorship of the accepting or rejecting claims is “KB Curator”,
who is privileged in terms of what s/he accepts or rejects being the
KB point of view at any given time.

You are essentially raising the question of evidence weighting or
credibility in an indirect way. REJECTED is a stronger form of
challengedBy and ACCEPTED is a stronger form of supportedBy, with
implied authorship “KB Curator”. I would not set up new predicates in
that way however.

---

Tim Clark, Ph.D.
Assistant Professor of Neurology, Harvard Medical School
Director of Informatics, MassGeneral Institute for Neurodegenerative
Disease
co-Director, Data and Statistics Core, Massachusetts Alzheimer
Disease Research Center
website: http://mindinformatics.org
mobile: +1 617-947-7098 fax: +1 617-213-5418

On Jul 1, 2015, at 5:16 PM, Samuel Rosko <notifications@github.com
mailto:notifications@github.com> wrote:

The current model does not include where evidence is rejected,
there's also no mp predicate for indicating that

—
Reply to this email directly or view it on GitHub
#15.

Richard D Boyce, PhD
Assistant Professor of Biomedical Informatics
Faculty, Center for Pharmaceutical Policy and Prescribing
Faculty, Geriatric Pharmaceutical Outcomes and Gero-Informatics Research and Training Program
University of Pittsburgh
rdb20@pitt.edu
Office: 412-648-9219
Twitter: @bhaapgh

[jodischneider]

One pending question that we have is how to deal with rejected evidence. We've been discussing using mp:Qualifer but it is an mp:Sentence whereas we need a graph.

So far the questions I have are:

• It seems right that a declarative statement cannot be a graph; do you agree?
• Are there any other use cases for mp:Qualifiers that aren't mp:Sentences?
• Would cause problems if mp:Qualifier were just an mp:Representation (rather than an mp:Sentence)?
• Are there any other problems/contraindications, using mp:Qualifier for the following use case?

General situation:

• Evidence (data/methods/materials) may be rejected as well as claims.
• Evidence may be rejected for some uses, while it is considered relevant for other uses
  • for example, a curator may state: "I reject the use of evidence item X to support or refute claim Y"
  • "Rejections are based on the explicit inclusion criteria plus the curator's judgement, and a reason for rejection is always required. Rejected evidence can apply to a single claim or a whole "family" of claims that share some commonality. For example, a poorly done in vitro experiment conducted to assess the metabolism of Drug X can be rejected for all "Drug X substrate-of Y" claims."​
• We need to record the reason for rejection, the curator who rejected an item, and the rejection date.
• Inclusion criteria are implemented in the evidence base. Belief criteria are implemented as a separate layer, which generates "multiple worlds" of knowledge bases.
  • This supports multiple groups of curators using the same evidence base to discuss rationales for different decisions ('boundary object').
• Rejection records several pieces of information that should be query-able:
  • the reason for rejection
  • who rejected it
  • when

So far, we've been looking at using mp:Qualifier to track rejection. The main problem seems to be that mp:Qualifier is a mp:Sentence (e.g. declarative statement). While we could make a sentence out of this, "Curator A rejected this for reason B at YYYY-MM-DDTHH:MM:SS-04:00" we need to record all that information separately (e.g. in a graph) so that it can be queried.