Skip to content

RO checklist evaluation API

Jump to bottom Edit New page
Soumya Brahma edited this page Feb 6, 2016 · 17 revisions

Table of Contents

API function overview

The checklist evaluation API is intended to provide access to the minim-based evaluation of Research Objects, used to test for completeness, executability, repeatability and other desired features. The functionality provided is based on the ro-manager evaluate checklist option: 

  ro evaluate checklist [ -d <dir> ] [ -a | -l <level> ] <minim> <purpose> [ <target> ]

where:

  • <dir> is the directory containing the RO to be evaluated
  • <level> indicates the level of information detail to be returned
  • <minim> is a URI reference for a minimum information model resource from which the checklist definition is obtained
  • <target> is a target resource with respect to which the evaluation is performed; the default <target> is the RO itself, but a component within the RO may be selected.
  • <purpose> is a keyword indicating the purpose for which the RO or <target> is to be evaluated.
For example: 
  ro evaluate checklist -d /workspace/myro -l all /workspace/minim.rdf "creation" myro/wfoutput.dat

might evaluate the RO at /workspace/myro using the minim model in file /workspace/minim.rdf.

The Web API is intended to provide remote access to the above functionality using simple HTTP requests.

Research Objects and other data are provided as web resources, and indicated in the API using their URIs.

API usage

Suppose we have:

A MINIM minimum information model description (containing checklist definitions including one for repeatability) at http://another.example.com/minim/repeatable.rdf The checklist definition in the MINIM model for repeatability is associated with the purpose keyword "repeatable". Note: there is an example of a simple minim model at https://github.com/wf4ever/ro-catalogue/blob/master/v0.1/simple-requirements/simple-requirements-minim.rdf

The checklist evaluation would then be invoked in a sequence of two HTTP operations:

  • Client retrieves service document:
C: GET /evaluate/checklist HTTP/1.1
C: Host: service.example.org
C: Accept: application/rdf+xml

S: HTTP/1.1 200 OK
S: Content-Type: application/rdf+xml
S:
S: <?xml version="1.0"?>
S: <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
S:   xmlns:roe="http://purl.org/ro/service/evaluate/">
S:   <rdf:Description rdf:about="">
S:     <roe:checklist>/evaluate/checklist{?RO,minim,target,purpose}</roe:checklist>
S:   </rdf:Description>
S: </rdf:RDF>
  • Client parses the service document, extracts the URI template for the checklist evaluation service and assembles URI for the desired evaluation result (cf. RFC6570), and issues a second HTTP GET request:
C: GET /evaluate/checklist?RO=http%3A%2F%2Fsandbox.example.org%2FROs%2Fmyro
         &minim=http%3A%2F%2Fanother.example.com%2Fminim%2Frepeatable.rdf
         &purpose=repeatable HTTP/1.1
C: Host: service.example.org
C: Accept: application/rdf+xml

S: HTTP/1.1 200 OK
S: Content-Type: application/rdf+xml
S:
S: <?xml version="1.0"?>
S: <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
S:   xmlns:...="..."
S:   >
S:   <rdf:Description rdf:about="...">
S:     (Result of checklist evaluation)
S:   </rdf:Description>
S: </rdf:RDF>
The result from the second request is the checklist evaluation result. The URI shown above has been split over several lines for readability - the actual HTTP request must present it without whitespace. The optional target URI parameter has been omitted in this example on the assumption that the target is the Research Object itself.

See also:

Link relations

http://purl.org/ro/service/evaluate/checklist

This relation is generally used used in the service description document

It indicates a relation between a service description and a URI template for RO evaluation results using the described service. The URI template is is used to construct a service result URI by:

  1. applying the URI template expansion procedures with caller-supplied RO URI, minim URI, purpose and target URIs, and
  2. resolving the resulting URI-reference to an absolute URI using normal URI resolution rules (e.g. typically, using the service document URI as a base URI)
See also:

HTTP methods

The service description is obtained in response to an HTTP GET to a checklist evaluation service URI.

The checklist evaluation service responds to an HTTP GET with the results of a checklist evaluation, using the URI defined by expanding the template provided by the service description.

Resources and formats

Service description

The checklist evaluation service description is an RDF file that contains URI templates for accessing RO related services, including checklist evaluation. The RDF syntax used may be content negotiated. In the absence of content negotiation, RDF/XML should be returned.

Example: 

<?xml version="1.0"?>
  <rdf:RDF
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:roe="http://purl.org/ro/service/evaluate/"
    >
    <rdf:Description rdf:about="">
      <roe:checklist>/evaluate/checklist{?RO,minim,target,purpose}</roe:checklist>
    </rdf:Description>
  </rdf:RDF>

Research Object

See:

Minim description

NOTE: this section describes the original Minim model structure, which is still fully supported by the software. However, the capabilities described here have been refactored and extended by a revised minim, model, which is described more fully in https://github.com/wf4ever/ro-manager/blob/develop/Minim/Minim-description.md and https://github.com/wf4ever/ro-manager/blob/develop/Minim/minim-revised.md

The MINIM description contains 3 levels of description:

  • minim:Checklist associates a target and purpose (e.g. runnable RO) to a minim:Model
to be evaluated.
  • minim:Model encodes the checklist (list of requirements) to be evaluated.
(There is provision for MUST / SHOULD / MAY requirements in a checklist to cater for limited variation in levels of conformance.)
  • minim:Requirement is a single requirement (checklist item), which is associated with a rule for evaluating whether or not it is satisfied or not satisfied. Each rule makes reference to a "checklist primitive" function. Additional capabilities can be added (in due course) by expanding the set of available checklist primitives (e.g. see http://www.wf4ever-project.org/wiki/display/docs/RO+decay+detection+using+checklists)
These 3 levels are called out in the examples that follow

See:

Minim Checklist

The "checklist" (previously called "constraint") describes a mapping from target and purpose values to a particular minim:Model to be used as the basis of an evaluation. Relative URI references are resolved relative to the location of the Minim resource. In this example, the Minim resource is taken from the root directory of an RO, so "." refers to the RO itself. 

<minim:Checklist rdf:about="#runnable_RO">
  <minim:forPurpose>Runnable</minim:forPurpose>
  <minim:onResource rdf:resource="." />
  <minim:toModel rdf:resource="#runnable_RO_model" />
  <rdfs:comment>
    Constraint to be satisfied if the RO is to be runnable
  </rdfs:comment>
</minim:Constraint>

Minim Model (check item list)

A minim model represents a list of check items to be evaluated. It enumerates of a number of requirements which may be declared at levels of MUST, SHOULD or MAY be satisfied for the model as a whole to be considered satisfied. This follows a structure for minimum information models proposed by Matthew Gamble. 

<minim:Model rdf:about="#runnable_RO_model">
  <rdfs:label>Runnable RO</rdfs:label>
  <rdfs:comment>
    This model defines information that must be available for the 
    requirements Research Object to be runnable.
  </rdfs:comment>
  <minim:hasMustRequirement rdf:resource="#environment-software/lpod-show" />
  <minim:hasMustRequirement rdf:resource="#environment-software/python" />
  <minim:hasMustRequirement rdf:resource="#isPresent/workflow-instance" />
  <minim:hasMustRequirement rdf:resource="#isPresent/workflow-inputfiles" />
</minim:Model>

Minim Requirements

Minim Requirements are evaluated using rules, which in turn invoke checklist evaluation primitives with appropriate parameters. This structure allows a relatively wide range of checklist items to be evaluated based on a relatively small number of primitive tests. The examples show the various primitives.

Requirement for an RO to contain a workflow primitive

The minim:ContentMatchRequirementRule is driven by a SPARQL query probe which is evaluated over a merge of all the RO annotations (including the RO manifest). In this case, it simply tests that the query can be satisfied. The minim:showpass and minim:showfail properties indicate strings that are used for reporting the status of the checklist evaluation. 

<!-- Workflow instance must be present -->
<minim:Requirement rdf:about="#isPresent/workflow-instance">
  <minim:isDerivedBy>
    <minim:ContentMatchRequirementRule>
      <minim:exists>
        ?wf rdf:type wfdesc:Workflow .
      </minim:exists>
      <minim:showpass>Workflow instance or template found</minim:showpass>
      <minim:showfail>No workflow instance or template found</minim:showfail>
      <minim:derives rdf:resource="#isPresent/workflow-instance" />
    </minim:ContentMatchRequirementRule>
  </minim:isDerivedBy>
</minim:Requirement>
Liveness testing

To test for liveness of a resource, the evaluator will need to attempt to access the resource. If it is a local file, a file existence check should suffice. If it is a web resource, then a success response to an HTTP HEAD request is expected. 

<!-- Workflow descriptions must be accessible (live) -->
<minim:Requirement rdf:about="#workflows_accessible">
  <minim:isDerivedBy>
    <minim:ContentMatchRequirementRule>
      <minim:forall>
        ?wf rdf:type wfdesc:Workflow .
      </minim:forall>
      <minim:isLiveTemplate>
        {+wf}
      </minim:isLiveTemplate>
      <minim:showpass>All declared workflow descriptions are accessible</minim:showpass>
      <minim:showfail>Workflow description %(wf)s is not accessible</minim:showfail>
      <minim:derives rdf:resource="#workflows_accessible" />
    </minim:ContentMatchRequirementRule>
  </minim:isDerivedBy>
</minim:Requirement>
This varies from the simple aggregation test in that the minim::aggregatesTemplate property is replaced by a minim:isLiveTemplate property.
Software environment testing

A minim:SoftwareEnvironmentRule tests to see if a particular piece of software is available by issuing a command and checking the response against a supplied regular expression. (This test is primarily intended for local use within RO-manager, and may be of limited use on the evaluation service as the command is issued on the host running the evaluation service, not on the host requesting the service.) 

<!-- Environment needs python -->
<minim:Requirement rdf:about="#environment-software/python">
  <minim:isDerivedBy>
    <minim:SoftwareEnvironmentRule>
      <minim:command>python --version</minim:command>
      <minim:response>Python 2.7</minim:response>
      <minim:show>Installed python version %(response)s</minim:show>
      <minim:derives rdf:resource="#environment-software/python" />
    </minim:SoftwareEnvironmentRule>
  </minim:isDerivedBy>
</minim:Requirement>

Proposed Minim enhancements

In viewing the proposals for liveness and integrity testing, compare with the current framework used for testing that an RO aggregates a specified resource; the following rule example tests that all workflow inputs are aggregated by the evaluated RO: 

<minim:ContentMatchRequirementRule>
  <minim:forall>
    ?wf rdf:type wfdesc:Workflow ;
        wfdesc:hasInput [ wfdesc:hasArtifact ?if ] .
  </minim:forall>
  <minim:aggregatesTemplate>{+if}</minim:aggregatesTemplate>
    :
</minim:ContentMatchRequirementRule>
Integrity testing

In this context, "integrity testing" means checking that a resource content matches some expected (e.g. previously calculated) value.

The integrity testing builds upon the liveness testing, by adding a reference to a resource with which the tested resource may be compared, by way of the minim:contentMatchTemplate property. Note that this property says nothing about using a hash or checksum value; the plan is that it's just the URI of a resource with which to compare. But that URI may be an ni: URI (cf. http://tools.ietf.org/html/draft-farrell-decade-ni-06), in which case, instead of trying to dereference the URI and do a byte-by-byte comparison, instead it calculates the appropriate hash function over the resource being tested and compares that with the value encoded by the ni: URI. To check for a known constant value, a data: URI (http://tools.ietf.org/html/rfc2397) can be used.

This structure assumes that appropriate annotations have been created to allow a SPARQL query to discover the URI of an appropriate resource with which to match the content. If may turn out that further queries against a separate RO are needed, in which case the above structure may need to be expanded. In the example below, the annotation is a roeval:contentMatch property directly about the resource being integrity-checked. 

<minim:ContentMatchRequirementRule>
  <minim:forall>
    ?wf rdf:type wfdesc:Workflow ;
        wfdesc:hasInput [ wfdesc:hasArtifact ?if ] .
    ?if roeval:contentMatch ?match .
  </minim:forall>
  <minim:accessTemplate>{+if}</minim:accessTemplate>
  <minim:contentMatchTemplate>{+match}</minim:contentMatchTemplate>
    :
</minim:ContentMatchRequirementRule>
Add a custom footer

RO Services and APIs

Clone this wiki locally