Test Harness

Prerequisites

• Install the project dependencies with yarn
• Generate the binary for your platform with yarn workspace @stoplight/spectral-cli build.binary. This will also compile the project from TS -> JS

Running the suite

Run yarn test.harness from your terminal

Running a selected tests

Test Harness uses Jest under the hood. You can use all CLI options that Jest supports: https://jestjs.io/docs/cli

You can run one or multiple tests by passing a path to test-harness command. All scenarios are converted to .js files under the ./tests directory. Hence you must use paths relative to the ./tests directory, like in the following example:

# path to scenario file: `test-harness/scenarios/require-module.scenario`
yarn test.harness test-harness/tests/require-module

Matching test files

All test files are matched using a glob **/*.scenario. This means that you can:

• nest test files in subdirectories
• skip files by suffixing name with .skip or some other suffix.

Adding a new test

• Create a new file in the ./scenarios directory. It can have any name and any extension, it does not really matter.
• Use the following template and put your stuff:

====test====
Test text, can be multi line.
====document====
some JSON/YAML document
====command====
{bin} lint --foo {document} --bar
====stdout====
expected output
====stderr====
optional/alternative to stdout for expected errors
====status====
(optional) expected exit code (`0` or `1`)

A real example?

====test====
scenario-example
====document====
openapi: 3.0.2
servers:
  - url: "localhost"
info:
  title: "my api"
  description: "An example"
  contact:
    name: "Stoplight"
  version: "1.0"
paths:
  /todos:
    get:
      tags:
        - "example"
      description: "Example endpoint"
      operationId: getTodos
      responses:
        200:
          description: Get Todo Items
          content:
            'application/json':
              example: hello
====command====
{bin} lint {document}
====stdout====
OpenAPI 3.x detected

{document}
 2:6  warning  some-rule  This rule is complaining about something.

✖ 1 problems (1 error, 0 warning, 0 infos, 0 hints)

Custom assets

Apart from {document}, you are allowed to provide any extra fixture your test may require. An example of such a fixture could be a ruleset, but also a you would like to output Spectral validation results to.

The syntax varies a bit from regular keywords and can be expressed in the following way asset:<pattern> where pattern is any string matching [A-Za-z0-9.\-] regular expression, for example asset:my-ruleset or asset:petstore.oas2.json.

A real example?

====test====
assets real-life example
====document====
openapi: 3.0.0
info:
  version: 1.0.0
  title: Stoplight
paths: {}
====asset:ruleset====
extends: spectral:oas
rules:
  oas3-api-servers: error
====command====
{bin} lint {document} -r {asset:ruleset}
====status====
1
====stdout====
OpenAPI 3.x detected

{document}
 1:1    error  oas3-api-servers  OpenAPI `servers` must be present and non-empty array.
 1:1  warning  openapi-tags      OpenAPI object should have non-empty `tags` array.
 2:6  warning  info-contact      Info object should contain `contact` object.
 2:6  warning  info-description  OpenAPI object info `description` must be present and non-empty string.

✖ 4 problems (1 error, 3 warnings, 0 infos, 0 hints)

Things to keep in mind when creating the files:

• 1 test per file, we do not support multiple splitting.
• Be precise with the separators. They should be 4 before AND after the word. ====
• The keywords are test, document, command, env, asset:<pattern>, status, stdout, and stderr, nothing else at the moment
• You can run all the tests on the same port 4010, but you can also choose another one
• You can pipe your command to grep. For example, using {bin} lint {document} | grep 'expected-rule-name' can be used to test for the presence of a specific violation. But, beware of being overly specific here.

Technical details

• A RegExp is used to split the content
• Temporary files with the document and/or any specified assets are stored on your disk
• Spectral gets spawned with the specified arguments and output is matched
• {document} and {asset:<pattern>} can be used in command, stdout or stderr, and is replaced with the full file path