Skip to content

TestResult Format Specifiation

Jump to bottom Edit New page
Krinkle edited this page Sep 9, 2014 · 1 revision

TestResult Format Specification Version 1

Editor's Draft 11 July 2012

This version
https://github.com/jquery/testswarm/wiki/TestResult-Report-Specification
Editors
Timo Tijhof

1. Introduction

This document specified the structure that a TestResult Report (hereafter referred to as "report") must follow. It is intended to be lightweight and usable for summarizing test results as generated by any compatible application for performing unit tests ("frameworks"). There are very few requirements of the report format, so it is expected to be compatible with most of the popular unit test frameworks in the world today.

Those that create or otherwise use reports (hereafter referred to as "user") can transport these reports between applications in a format of their choosing (as long as it supports nested key/value pairs, where a key can be any string, and a value a number, boolean, string, array or object). The recommended data-interchange format is JSON (see also www.json.org / RFC 4627). This document will use JSON in examples. Inside an application flow the user should use a native language structure if possible (e.g. objects in JavaScript, associative arrays in PHP, etc.).

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

2. Structures

The report must have exactly one top-most group of key/value pairs (hereafter referred to as "objects" having "properties") that follows the specification for the "Root" structure, as specified in section 2.1 below.

All of these objects specified below have one common requirement. They must have all of the described properties. Additional properties not specified below are not permitted. And, with the exception of those marked as "optional", none of the properties specified below may be permitted.

2.1 Root

Both the groups and assertions properties are optional. This allows for usage for simple linear test suites, as as well test suites that have a deeply nested hierarchy. Both may be present simultaneously, which indicates that some assertions are not contained in a group. Though both are optional, the root must contain at least one of groups or assertions.

2.1.1 name

  • Required
  • Content: Name String

2.1.2 summary

  • Required
  • Content: Summary Object

2.1.3 time

  • Optional
  • Content: Number

The total run-time of all assertions in this test suite (in milliseconds). This sum should match the recursively calculated sum of all assertion times. However it does not have to match that sum. The root time may include execution time between assertions (like a setup or teardown for a group or the test suite as a whole) if a user chooses to do so.

2.1.4 groups

  • Optional
  • Content: Array containing Group objects

2.1.5 assertions

  • Optional
  • Content: Array containing Assertion objects

2.2 Summary

The values of a summary object should reflect the sum of all containing groups and/or assertions (recursively).

2.2.1 total

  • Required
  • Content: Number

Total number of assertions in this group. Should be recursive to include nested assertions if the group contains other groups.

2.2.2 failed

  • Required
  • Content: Number

Number of assertions that are to be considered "failing the test suite". The user's framework may have different types of unsuccessful results. It is up to the user to decide whether or not to include a non-passing result in the this total.

For example, in PHPUnit, there are annotations to indicate known "Broken" or "Incomplete" tests, which are skipped during execution or other wise excluded from the "failure" count.

2.3 Group

2.3.1 name

  • Required
  • Content: Name String

2.3.2 summary

  • Required
  • Content: Summary Object

2.3.3 time

  • Optional
  • Content: Number

Total run-time of all containing assertions (in milliseconds). May or may not include execution time between, before and/or after assertions within the boundaries of the group.

2.3.4 groups

  • Optional
  • Content: Array containing Group objects

2.3.5 assertions

  • Optional
  • Content: Array containing Assertion objects

2.4 Assertion

2.4.1 name

  • Required
  • Content: Name String

2.4.2 source

  • Optional
  • Content: Source Object

2.4.3 time

  • Optional
  • Content: Number

Run-time in milliseconds.

2.4.4 status

  • Required
  • Content: String

Summarizes the outcome of the assertion. Must be one of:

  • pass: The assertion passed as expected
  • fail: The assertion failed to meet the expected outcome

No non-standard values allowed here.

2.4. result

  • Optional
  • Content: Result Object

2.5 Source

2.6 Result

2.7 Name

Free form text. Title representing the object the property belongs to. Must not be used for unique identification. May contain some kind of identification for human readers to identify a certain version of build, but does not have to.

For example, a test suite may be named like:

  • "ACME Project Test Suite"
  • "ACME 2.0-beta2 Test Suite"
  • "ACME 2.0 Test Suite (04708c0)"

And similarly there may be multiple assertions with the same name. For example there could be a group of tests that tests an object in different ways after performing a certain operation. Then another test group could run those same group of assertion when doing a different operation:

  • Group: FooBar applies quuxification to x
  • Assert: fb.quuxified matches expected state
  • Assert: ...
  • Group: FooBaz does not quuxify y
  • Assert: fb.quuxified matches expected state
  • Assert: ...
Add a custom footer
Add a custom sidebar
Clone this wiki locally