As the title says, ze plan is to implement an all platform compatible utility, which takes specked cucumber event stream from stdin and spits output to stdout.
This would enable any cucumber implementation to implement a non complicated common event stream formatter, which can be streamed to this tool to get the actual pretty, junit... format output.
There are many other ideas how this could extend, but their are not clear yet, like a common test suite for all implementations.
Formatter error output should be stderr.
So far it is only a proof of concept and events have no clear specification.
Currently implemented formatters:
- progress
The formatter will be shipped as a binary for all OS architectures
separately. A cucumber implementation could advice user to install it with
a single one liner command so it is available in PATH or install it
together with a cucumber implementation.
If cucumber implementation see this binary in PATH then it could
stream events through it and output specific format to the stdout or other
output stream.
In order to be able to print results in all different kind of formats. General purpose formatter, expects these events passed in as a stream of json objects.
TestRunStartedTestSourceStepDefinitionFoundTestCaseStartedTestStepStartedTestStepFinishedTestCaseFinishedTestRunFinishedTestAttachment
Triggers when tests are started. Specifies protocol version.
{
"event": "TestRunStarted",
"version": "0.1.0",
"timestamp": 1461436176456,
"suite": "main"
}event- name of event.version- (optional) protocol version used for events. If not provided, latest stable protocol version is expected.timestamp- unix timestamp in milliseconds since epoch. When the test run started.suite- (optional) name of the test suite.
When a test source is parsed, this event should be sent with plain text of
source. It will be determined by source extension found in location.
Currently only gherkin source is supported.
{
"event": "TestSource",
"location": "features/simple.feature:1",
"source": "Feature:\n Scenario: passing\n Given passes"
}event- name of event.location- location in source file, based on pattern{path}:{line}.source- is plain text of test source.
Should fire when step regexp or other matching algorithm determines step implementation in the source code.
Note: There may be ambiguous matches.
{
"event": "StepDefinitionFound",
"location": "features/simple.feature:5",
"definition": "FeatureContext::passing():6",
"arguments": [
[12, 18],
[23, 26]
]
}event- name of event.location- location in source file, based on pattern{path}:{line}.definition- reference to step definition in source code.arguments- list of positions for arguments which were matched. Positions are determined on step text step keyword should be omitted when calculating argument positions.
Should fire when starting to execute scenario or scenario outline example.
{
"event": "TestCaseStarted",
"location": "features/simple.feature:4",
"timestamp": 1461436176456
}event- name of event.location- location in source file, based on pattern{path}:{line}.timestamp- unix timestamp in milliseconds since epoch (when the test case started).
Should fire right before step execution.
{
"event": "TestStepStarted",
"location": "features/simple.feature:5",
"timestamp": 1461436176456
}event- name of event.location- location in source file, based on pattern{path}:{line}.timestamp- unix timestamp in milliseconds since epoch (when the test step started).
Should fire right after step has finished execution and give appropriate status and details.
{
"event": "TestStepFinished",
"location": "features/simple.feature:5",
"status": "failed",
"timestamp": 1461436176456,
"summary": "error - user was not found by id: 1",
"details": "error details\ndebug information"
}event- name of event.location- location in source file, based on pattern{path}:{line}.status- can be one ofpassed,failed,skipped,pending,undefined,ambiguous.timestamp- unix timestamp in milliseconds since epoch (when the test step finished).summary- (optional) one line summary for step result.details- (optional) multi-line detailed description of step result.
Should fire after all steps are executed for scenario or outline example. Should provide appropriate result status.
{
"event": "TestCaseFinished",
"location": "features/simple.feature:5",
"status": "failed",
"timestamp": 1461436176456
}event- name of event.location- location in source file, based on pattern{path}:{line}.status- can be one ofpassed,failed,skipped,pending,undefined,ambiguous.timestamp- unix timestamp in milliseconds since epoch (when the test case finished).
Should fire after all tests are run, or if “stop on failure” flag was specified and failure occurred. It should carry the status of all tests and resource usage summary information.
{
"event": "TestRunFinished",
"status": "failed",
"timestamp": 1461436176456,
"memory": 3456765,
"snippets": "implement undefined steps with the following snippets:"
}event- name of event.status- can be one ofpassed,failed,skipped,pending,undefined,ambiguous.timestamp- unix timestamp in milliseconds since epoch (when the test run finished).memory- (optional) memory consumption in bytes used by all tests.snippets- (optional) undefined step implementation source code snippets.
An attachment to test cases, for example a screenshot or video. Might be exception stack traces.
{
"event": "TestAttachment",
"location": "features/simple.feature:5",
"mime": "image/png",
"data": "YWJjZGU=",
"encoding": "base64",
"timestamp": 1461436176456
}event- name of event.location- location in source file, based on pattern{path}:{line}.mime- mime type of given media file.data- encoded data.encoding- data must be encoded to transfer with json format, usually base64 or base85.timestamp- unix timestamp in milliseconds since epoch (when the attachment was created).