Features

Overrides

When creating a Scenario, one can override the properties defined in the Action Templates easily by just overriding the same keys:

Example 1: overriding action properties

### /actions/create-user.yaml
type: REST
service: my-service
endpoint: /users
method: POST
data:
  firstname: Bill
  lastname: Gates

### /scenarios/creating-test-user.yaml
description: ...
actions:
  - create-user
    data:
      firstname: George

The above scenario would execute the create-user action by sending following payload to my-service/users:

{
  "firstname": "George",
  "lastname": "Gates
}

All properties defined in an action template can be overridden by the scenario. Moreover additional parameters can be defined in a scenario which would be added to the execution of the action (e.g. above scenario could define additional payload attributes like age or define own response validation rules).

Variables

It is possible to set variables which can be used in multiple actions. Variables have a scope of a scenario and can be set in multiple ways:

Example 2: setting variables for a scenario statically

description: "test description"
variables:
  test-user: 1371
actions:
  - name: do-something
    method: POST
    data:
      userId: {{test-user}}

Here a variable test-user is being set for the scope of the whole scenario and it can be referenced in any action by {{test-user}}.

Example 3: setting variables dynamically

description: "test description"
actions:
  - name: create-user
    data:
      name: "Alice"
      job: "Teacher"
    variables:
      aliceId: "res.id"
  - name: activate-user
    data:
      userId: {{aliceId}}

This time, a variable aliceId is being created & set at runtime after successful execution of crate-user action. The value is going to be retrieved directly from the response (referenced ALWAYS by res) - assuming the payload contains an id field. All subsequent actions (which execute after create-user) can reference the value by {{aliceId}}. This is possible because variables will be resolved at runtime (e.g. when the payload of the activate-user action is being evaluated for the REST call).

Example 4: setting evaluated variables

description: "test description"
actions:
  - name: do-something
    method: POST
    data:
      timestamp: <<< Date.now() >>>

It is possible to create/use variables which are evaluated at runtime. Hereby all NodeJS statements are valid. The return type of <<< >>> will be number, while it is also possible to create strings with {{{ }}}.

Response Validation

For REST actions it is possible to send validation rules, which should be applied on the response payload:

Example 5: applying response validation

...
actions:
  - name: search-for-user
    data:
      nameLike: "john"
    responseValidation:
    - "res.length >== 0"
    - "res[0].lastName !== null"
...

All rules specified under responseValidation will be applied to the payload coming from the REST call. The data itself can be referenced by either res (response body) or head (response headers) variables. If one of the rules fails, the whole action and thus the wrapping scenario will count as failed!

Flow control

When invoking a scenario, all defined action are being executed in natural order (top to bottom). There are however multiple features which help to control that execution flow:

Imports

If the setup stage for the test needs multiple actions, it's far more simple to define such actions in a separate scenario, which can be imported in different test scenarios:

s0-before.yaml

description: "test prepararation"
actions:
  - name: before-test
    method: POST

s26-test-something.yaml

description: "test X"
import:
  - s0-before
actions:
  - name: test-something
    method: POST
    data:
      ...

Now, the actual test file can shrink and we only have to concentrate on the business logic of the test. All the setup (creating user, authenticating, etc.) can now be put into single scenario and be reused in multiple different tests.

If we want to reuse scenarios in the tearDown phase, we can use after in the same way as import - listing all scenario names that should be executed after the actual test actions:

description: "test description"
actions:
  - name: do-something
  - name: do-more
after:
  - s0-after

Invoke On Fail

Usually all actions of a scenario are executed until either all succeed or one failed. On failure the whole execution is being stopped and the process exits with errorCode=1. Sometimes however it is important to do some actions even though the test failed - e.g. delete the created user. In such cases each action has an attribute ìnvokeEvenOnFail which defaults to false but can be overridden in a scenario to force the execution of that action:

description: "test description"
actions:
  - name: create-test-user
    ...
  - name: do-what-users-do
    ...
  - name: delete-user
    invokeEvenOnFail: true
    ...

Now, even when the do-what-users-do action fails to execute properly, the created test user will be deleted!

Allow failure

Usually if an action fails the process exits with errorCode=1. With the flag allowFailure an action can be marked that a failure of this particular action does not contribute to the error code of the process. The flag is optional and defaults to false.

description: "test description"
actions:
  - name: create-test-user
    ...
  - name: do-what-users-do
    ...
  - name: delete-user
    ...
  - name: cleanup-user
    invokeEvenOnFail: true
    allowFailure: true
    ...

The cleanup-user action is always invoked because of the invokeEvenOnFail flag to ensure that the user has been deleted at the end of the scenario even if e.g. the do-what-users-do action did fail. However if all actions succeed the user has already been deleted and the cleanup-user action will fail. To prevent the scenario from also failing the allowFailure flag is used.