Skip to content

Latest commit

 

History

History
381 lines (319 loc) · 14.6 KB

CONTRIBUTING.md

File metadata and controls

381 lines (319 loc) · 14.6 KB
Raw

Annotation-model: Guidelines for Contributing Tests

This document describes the method people should use for authoring tests and integrating them into the repository. Anyone is welcome to submit new tests to this collection. If you do, please create the tests following the guidelines below. Then submit them as a pull request so they can be evaluated

Structure

Tests are organized by major section of the Annotation Model specification. The folders associated with these are:

  • annotations
  • bodiesTargets
  • collections
  • specificResources
    • selectors
    • states

Within these folders, special files ending with the suffix ".test" provide the source for the test as a set of declarative assertions about the required shape of the conforming JSON object. These files are transformed using a test generation tool into ".html" files that are then accessed by the Web Platform Test framework.

There are a few other folders that provide supporting materials for the tests:

  • common - assertionObjects, conditionObjects, and other supporting materials
  • definitions - JSON Schema definitions that can be referenced
  • scripts - JavaScript that are included by tests
  • tools - supporting scripts and files

NOTE: The files in the definitions folder are expected to be JSON Schema definitions - basically commonly used concepts that are referenced by other JSON Schema files in the system. All of these 'definitions' are preloaded by the system before any other parts of a test are processed.

Test Cases

Each test is expressed as a simple (or complex) requirement in a test file. For each section of the document, the requirement is represented as a structure that describes the nature of the test, and then includes or references minimal JSON Schema that test the assertions implied by the requirement.

The structure of a test case is defined using a JSON-LD Context. That context defines the following terms:

Keyword Values Meaning
name string The name of this test for display purposes
description string A long self-describing paragraph that explains the purpose of the test and the expected input
ref URI An optional reference to the portion of the specification to which the test relates
testType automated, manual, ref The type of test - this informs WPT how the test should be controlled and presented
skipFailures list of strings An optional list of assertionType values that, if present, should have their test skipped if the result would be "unexpected". Defaults to the empty list.
assertions list of URI, List @@@ATRISK@@@, or AssertionObject The ordered collection of tests the input should be run against. See JSON Schema Usage for the structure of the objects. URI is relative to the top level folder of the test collection if it has a slash; relative to the current directory if it does not. @@@@ATRISK@@@@ Lists can be nested to define groups of sub-tests. Assertions / groups can be conditionally skipped. See Assertion Lists for more details.
content URI or object An object containing content to be checked against the referenced assertions, or a URI from which to retrieve that content

Each test case has a suffix of .test and a shape like:

{
  "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
  "name": "Verify annotation conforms to the model",
  "description": "Supply an example annotation that conforms to the basic structure.",
  "ref": "https://www.w3.org/TR/annotation-model/#model",
  "testType": "manual",
  "assertions": [
    "common/has_context.json",
    "common/has_id.json",
    {
      "$schema": "http://json-schema.org/draft-04/schema#",
      "title": "Verify annotation has target",
      "assertionType": "must",
      "expectedResult": "valid",
      "errorMessage": "The object was missing a required 'target' property",
      "type": "object",
      "properties": {
        "target": {
          "anyOf": [
          {
            "type": "string"
          },
          {
            "type": "array",
            "anyOf": [
            {
              "type": "string"
            }
            ]
          }
          ],
            "not": {"type": "object"}
        }
      },
      "required": ["target"]
    }
  ]
}

External references are used when the "assertion" is a common one that needs to be checked on many different test cases (e.g., that there is an @context in the supplied annotation).

NOTE: The title property of an assertionObject can contain markdown. This can help improve readability of the rendered assertions and debugging output.

NOTE: The content property does not yet have a defined use. One potential use would be to act as a pointer to a URI that can supply annotations from an implementation. In that case the URI would take a parameter with the test name as a way of telling the end point what test is running so it can deliver the right content.

Assertion Lists

The assertion list is an ordered list of assertions that will be evaluated against the submitted content. The list is required, and MUST have at least one entry. Entries in the list have the following types:

  • AssertionObject

An in-line Object as defined in the section Assertion Objects.

  • URI

A relative or absolute URI that references a AssertionObject in a .json file. If the URI is relative but contains no slashes, then it is considered to be in the current directory. If the URI is relative, contains slashes, but does not start with a slash then it is considered relative to the top of the tree of the current test collection (e.g., annotation-model).

  • List @@@ATRISK@@@

A nested Assertion List. While nested Assertion Lists are optional, if one is present it MUST have at least one entry. Entries are as in this list. Assertion Lists can be nested to any depth (but don't do that - it would be too hard to maintain).

Assertion Objects

In this collection of tests, Assertion Objects can be contained inline in the .test files or contained in external files with the suffix .json. The vocabularly and structure is as defined in JSON Schema v4 augmented with some additional properties defined in this section.

In general each JSON Schema definition provided in this test suite should be as minimal as possible. This promotes clarity and increases the likelihood that it is correct. While it is ---possible--- to create JSON Schema files that enforce many different requirements on a data model, it increases the complexity and can also reduce the atomicity of tests / sub-tests (because a test ends up testing more than one thing). Please try to avoid creating complex JSON Schema. (A notable exception is the situation where multiple properties of a structure are interdependent.)

Tools such as the JSON Schema Creator may be helpful in creating schema snippets that can be integrated into JSONtest Assertion Objects. Remember that the JSON Schema you create should be as permissive as possible to address the various shapes that a give property might take (e.g., a 'foo' might be a string or an object that contains sub-properties that express the string, or an array of 1 or more objects with those sub-properties).

In addition to the validation keys defined in JSON Schema v4, Schema files in this collection are also permitted to use the following keywords:

Keyword Values Meaning
onUnexpectedResult failAndContinue, failAndSkip, failAndAbort, passAndContinue, passAndSkip, passAndAbort Action to take when the result is not as expected. Default is failAndContinue
assertionType must, may, should Informs the system about the severity of a failure. The default is must
assertionFile URI An external file that contains an assertion SCHEMA. When this value is supplied, and local properties will override the ones loaded from the external file.
errorMessage string A human readable explanation of what it means if the test fails.
expectedResult valid, invalid Tells the framework whether validating against this schema is expected to succeed or fail. The default is valid

Example Assertion Object

{
  "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Verify annotation has @context",
    "type": "object",
    "expectedResult" : "valid",
    "properties": {
      "@context": {
        "anyOf": [
        {
          "type": "string"
        },
        {
          "type": "array",
          "anyOf": [
          {
            "type": "string"
          }
          ]
        }
        ],
          "not": {"type": "object"}
      }
    },
    "required": ["@context"]
}

Note that in the case where a feature is optional the JSON Schema MUST be crafted such that if the attribute is permitted to be missing from the content (so that the result is true), but when the attribute is present in the content it conforms to any requirements.

Condition Objects

A Condition Object is a sub-class of an Assertion Object. It allows the specification of the evaluation strategy for the assertions referenced by the object. An object is a Condition Object IFF it has a assertions property. In this case, there MUST NOT be an assertionFile property.

Keyword Values Meaning
compareWith and, or How should the result of any referenced assertions be compared. Defaults to and. Note that this implies there is also an assertions property with a nested list of assertions to compare.
assertions a list of assertions as in a Test Case above. This is required if there is a compareWith property

An example of a test that would pass if there were an @context OR there were an @id:

{
  "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
    "name": "A test that has an 'or' clause",
    "description": "A complex test that uses or-ing among a list of assertions",
    "ref": "https://www.w3.org/TR/annotation-model/#model",
    "testType": "manual",
    "assertions": [
    { "$schema": "http://json-schema.org/draft-04/schema#",
      "title": "must have context or id",
      "description": "A more complex example that allows one of many options to pass",
      "assertions": [
      { "title": "Condition Object",
        "description": "A pseudo-test that will get a result from the aggregate of its children",
        "assertionType": "must",
        "expectedResult": "valid",
        "errorMessage": "Error: None of the various options were present",
        "compareWith": "or",
        "assertions": [
          "common/has_context.json",
        "common/has_id.json"
        ]
      }
      ]
    }
    ]
}

Command Line Tools

Building the Test Files

The actual .html test case files are generated using the script tools/make_tests.py. This script will search the directory heriarchy looking for files ending on .test and creating .html files from them using the template in the tools folder. If you want to regenerate the examples too, supply the --examples option to the script.

Note that when submitting tests to the repository, the .html versions must be included.

Testing the Tests

Driving Tests with Input Files

Complex Examples

This section is a collection of more complex examples to illustrate the expressive power of the Assertion List structure. These can be used as templates for creating actual .test files.

Including and Overriding an Assertion

Assertions can be contained in external .json files. It is possible for an object in an Assertion List to include the external file and override one or more of its properties:

{
  "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
    "name": "Permit no target property",
    "description": "Ensure there is no 'target' property when there is a 'randomName' property in the Annotation",
    "assertions": [
    {
      "$schema": "http://json-schema.org/draft-04/schema#",
      "title": "Verify annotation has randomName",
      "type": "object",
      "properties": {
        "randomName": {
          "type": "string"
        }
      },
      "required": ["randomName"]
    },
    { "assertionFile" : "common/target.json",
      "title" : "Require target to be missing",
      "expectedResult" : "invalid",
      "errorMessage" : "The target MUST not be present when 'randomName' is also present",
    }
    ]
}

Nested Assertion Collections with Skip

Assertion Lists can be nested within Assertion Lists. This feature, combined with the onUnexpectedResult property, makes it possible to skip a collection of tests when an assertion in the list is not satisfied:

{
  "@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
    "name": "If there is no 'target' property, skip some tests",
    "description": "When 'target' is not present, other properties related to 'target' are not required",
    "assertions": [
      "common/context.json",
    [
    { "assertionFile" : "common/target.json",
      "errorMessage" : "Target was not present so skip the rest of this section",
      "onUnexpectedResult" : "failAndSkip"
    },
    "sometest.json",
    "sometest2.json",
    "sometest3.json"
    ]
    ]
} ;

Assertion that finds a specific @context Value

Sometimes you want a property to be flexible, but to have one and only one of a specific value. This is especially true with, for example, @context in JSON-LD. One way you might do this is:

{
  "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Verify a specific @context",
    "type": "object",
    "expectedResult" : "valid",
    "properties": {
      "@context": {
        "anyOf": [
        {
          "type": "string"
            "enum": [ "http://www.w3.org/ns/anno.jsonld" ]
        },
        {
          "type": "array",
          "minitems": "1",
          "uniqueItems": true,
          "additionalItems": true,
          "items" : [
          { "type": "string",
            "enum": [ "http://www.w3.org/ns/anno.jsonld" ]
          }
          ]
        }
        ],
          "not": {"type": "object"}
      }
    },
    "required": ["@context"]
}