Skip to content
This repository has been archived by the owner on Nov 8, 2024. It is now read-only.

Latest commit

 

History

History
861 lines (592 loc) · 32.1 KB

how-to-guides.rst

File metadata and controls

861 lines (592 loc) · 32.1 KB

How-To Guides

In the following guides you can find tips and best practices how to cope with some common tasks. While searching this page for particular keywords can give you quick results, reading the whole section should help you to learn some of the Dredd’s core concepts and usual ways how to approach problems when testing with Dredd.

Isolation of HTTP Transactions

Requests in the API description usually aren’t sorted in order to comply with logical workflow of the tested application. To get the best results from testing with Dredd, you should ensure each resource action (`API Blueprint`_) or operation (`OpenAPI 2`_) is executed in isolated context. This can be easily achieved using :ref:`hooks <hooks>`, where you can provide your own setup and teardown code for each HTTP transaction.

You should understand that testing with Dredd is an analogy to unit tests of your application code. In unit tests, each unit should be testable without any dependency on other units or previous tests.

Example

Common case is to solve a situation where we want to test deleting of a resource. Obviously, to test deleting of a resource, we first need to create one. However, the order of HTTP transactions can be pretty much random in the API description.

To solve the situation, it’s recommended to isolate the deletion test by :ref:`hooks <hooks>`. Providing before hook, we can ensure the database fixture will be present every time Dredd will try to send the request to delete a category item.

API Blueprint

FORMAT: 1A

# Categories API

## Categories [/categories]

### Create a Category [POST]
+ Response 201

## Category [/category/{id}]
+ Parameters
    + id: 42 (required)

### Delete a Category [DELETE]
+ Response 204

## Category Items [/category/{id}/items]
+ Parameters
    + id: 42 (required)

## Create an Item [POST]
+ Response 201

To have an idea where we can hook our arbitrary code, we should first ask Dredd to list all available transaction names:

$ dredd api-description.apib http://127.0.0.1:3000 --names
info: Categories > Create a category
info: Category > Delete a category
info: Category Items > Create an item

Now we can create a hooks.js file. The file will contain setup and teardown of the database fixture:

hooks = require('hooks');
db = require('./lib/db');

beforeAll(function() {
  db.cleanUp();
});

afterEach(function(transaction) {
  db.cleanUp();
});

before('Category > Delete a Category', function() {
  db.createCategory({id: 42});
});

before('Category Items > Create an Item', function() {
  db.createCategory({id: 42});
});

OpenAPI 2

swagger: "2.0"
info:
  version: "0.0.0"
  title: Categories API
  license:
    name: MIT
host: www.example.com
basePath: /
schemes:
  - http
consumes:
  - application/json
produces:
  - application/json
paths:
  /categories:
    post:
      responses:
        200:
          description: ""
  /category/{id}:
    delete:
      parameters:
        - name: id
          in: path
          required: true
          type: string
          enum:
            - "42"
      responses:
        200:
          description: ""
  /category/{id}/items:
    post:
      parameters:
        - name: id
          in: path
          required: true
          type: string
          enum:
            - "42"
      responses:
        200:
          description: ""

To have an idea where we can hook our arbitrary code, we should first ask Dredd to list all available transaction names:

$ dredd api-description.yml http://127.0.0.1:3000 --names
info: /categories > POST > 200 > application/json
info: /category/{id} > DELETE > 200 > application/json
info: /category/{id}/items > POST > 200 > application/json

Now we can create a hooks.js file. The file will contain setup and teardown of the database fixture:

hooks = require('hooks');
db = require('./lib/db');

beforeAll(function() {
  db.cleanUp();
});

afterEach(function(transaction) {
  db.cleanUp();
});

before('/category/{id}', function() {
  db.createCategory({id: 42});
});

before('/category/{id}/items', function() {
  db.createCategory({id: 42});
});

Testing API Workflows

Often you want to test a sequence of steps, a scenario, rather than just one request-response pair in isolation. Since the API description formats are quite limited in their support of documenting scenarios, Dredd probably isn’t the best tool to provide you with this kind of testing. There are some tricks though, which can help you to work around some of the limitations.

Note

`API Blueprint`_ prepares direct support for testing and scenarios. Interested? Check out :ghissue:`api-blueprint#21`!

To test various scenarios, you will want to write each of them into a separate API description document. To load them during a single test run, use the :option:`--path` option.

For workflows to work properly, you’ll also need to keep shared context between individual HTTP transactions. You can use :ref:`hooks <hooks>` in order to achieve that. See tips on how to :ref:`pass data between transactions <sharing-data-between-steps-in-request-stash>`.

API Blueprint Example

Imagine we have a simple workflow described:

FORMAT: 1A

# My Scenario

## POST /login

+ Request (application/json)

        {"username": "john", "password": "d0e"}


+ Response 200 (application/json)

        {"token": "s3cr3t"}

## GET /cars

+ Response 200 (application/json)

        [
            {"id": "42", "color": "red"}
        ]

## PATCH /cars/{id}
+ Parameters
    + id: 42 (string, required)

+ Request (application/json)

        {"color": "yellow"}

+ Response 200 (application/json)

        {"id": 42, "color": "yellow"}

Writing Hooks

To have an idea where we can hook our arbitrary code, we should first ask Dredd to list all available transaction names:

$ dredd api-description.apib http://127.0.0.1:3000 --names
info: /login > POST
info: /cars > GET
info: /cars/{id} > PATCH

Now we can create a hooks.js file. The code of the file will use global stash variable to share data between requests:

hooks = require('hooks');
db = require('./lib/db');

stash = {}

// Stash the token we've got
after('/login > POST', function (transaction) {
  stash.token = JSON.parse(transaction.real.body).token;
});

// Add the token to all HTTP transactions
beforeEach(function (transaction) {
  if (stash.token) {
    transaction.request.headers['X-Api-Key'] = stash.token
  };
});

// Stash the car ID we've got
after('/cars > GET', function (transaction) {
  stash.carId = JSON.parse(transaction.real.body).id;
});

// Replace car ID in request with the one we've stashed
before('/cars/{id} > PATCH', function (transaction) {
  transaction.fullPath = transaction.fullPath.replace('42', stash.carId)
  transaction.request.uri = transaction.fullPath
})

OpenAPI 2 Example

Imagine we have a simple workflow described:

swagger: "2.0"
info:
  version: "0.0.0"
  title: Categories API
  license:
    name: MIT
host: www.example.com
basePath: /
schemes:
  - http
consumes:
  - application/json
produces:
  - application/json
paths:
  /login:
    post:
      parameters:
        - name: body
          in: body
          required: true
          schema:
            type: object
            properties:
              username:
                type: string
              password:
                type: string
      responses:
        200:
          description: ""
          schema:
            type: object
            properties:
              token:
                type: string
  /cars:
    get:
      responses:
        200:
          description: ""
          schema:
            type: array
            items:
              type: object
              properties:
                id:
                  type: string
                color:
                  type: string
  /cars/{id}:
    patch:
      parameters:
        - name: id
          in: path
          required: true
          type: string
          enum:
            - "42"
        - name: body
          in: body
          required: true
          schema:
            type: object
            properties:
              color:
                type: string
      responses:
        200:
          description: ""
          schema:
            type: object
            properties:
              id:
                type: string
              color:
                type: string

Writing Hooks

To have an idea where we can hook our arbitrary code, we should first ask Dredd to list all available transaction names:

$ dredd api-description.yml http://127.0.0.1:3000 --names
info: /login > POST > 200 > application/json
info: /cars > GET > 200 > application/json
info: /cars/{id} > PATCH > 200 > application/json

Now we can create a hooks.js file. The code of the file will use global stash variable to share data between requests:

hooks = require('hooks');
db = require('./lib/db');

stash = {}

// Stash the token we've got
after('/login > POST > 200 > application/json', function (transaction) {
  stash.token = JSON.parse(transaction.real.body).token;
});

// Add the token to all HTTP transactions
beforeEach(function (transaction) {
  if (stash.token) {
    transaction.request.headers['X-Api-Key'] = stash.token
  };
});

// Stash the car ID we've got
after('/cars > GET > 200 > application/json', function (transaction) {
  stash.carId = JSON.parse(transaction.real.body).id;
});

// Replace car ID in request with the one we've stashed
before('/cars/{id} > PATCH > 200 > application/json', function (transaction) {
  transaction.fullPath = transaction.fullPath.replace('42', stash.carId)
  transaction.request.uri = transaction.fullPath
})

Making Dredd Validation Stricter

API Blueprint or OpenAPI 2 files are usually created primarily with documentation in mind. But what’s enough for documentation doesn’t need to be enough for testing.

That applies to both `MSON`_ (a language powering API Blueprint’s :apib:`Attributes <def-attributes-section>` sections) and `JSON Schema`_ (a language powering the OpenAPI 2 format and API Blueprint’s :apib:`Schema <def-schema-section>` sections).

In following sections you can learn about how to deal with common scenarios.

Avoiding Additional Properties

If you describe a JSON body which has attributes name and size, the following payload will be considered as correct:

{"name": "Sparta", "size": 300, "luck": false}

It’s because in both `MSON`_ and `JSON Schema`_ additional properties are not forbidden by default.

Requiring Properties

If you describe a JSON body which has attributes name and size, the following payload will be considered as correct:

{"name": "Sparta"}

It’s because properties are optional by default in both `MSON`_ and `JSON Schema`_ and you need to explicitly specify them as required.

Validating Structure of Array Items

If you describe an array of items, where each of the items should have a name property, the following payload will be considered as correct:

[{"name": "Sparta"}, {"title": "Athens"}, "Thebes"]

That’s because in `MSON`_, the default behavior is that you are specifying what may appear in the array.

Validating Specific Values

If you describe a JSON body which has attributes name and size, the following payload will be considered as correct:

{"name": "Sparta", "size": 42}

If the size should be always equal to 300, you need to specify the fact in your API description.

Integrating Dredd with Your Test Suite

Generally, if you want to add Dredd to your existing test suite, you can just save Dredd configuration in the dredd.yml file and add call for dredd command to your task runner.

There are also some packages which make the integration a piece of cake:

To find more, search for dredd in your favorite language’s package index.

Continuous Integration

It’s a good practice to make Dredd part of your continuous integration workflow. Only that way you can ensure that application code you’ll produce won’t break the contract you provide in your API documentation.

Dredd’s interactive configuration wizard, dredd init, can help you with setting up dredd.yml configuration file and with modifying or generating CI configuration files for `Travis CI`_ or `CircleCI`_.

If you prefer to add Dredd yourself or you look for inspiration on how to add Dredd to other continuous integration services, see examples below. When testing in CI, always pin your Dredd version to a specific number and upgrade to newer releases manually.

.circleci/config.yml Configuration File for `CircleCI`_

version: 2
jobs:
  build:
    docker:
      - image: circleci/node:latest
    steps:
      - checkout
      - run: npm install dredd@x.x.x --global
      - run: dredd apiary.apib http://127.0.0.1:3000

.travis.yml Configuration File for `Travis CI`_

before_install:
  - npm install dredd@x.x.x --global
before_script:
  - dredd apiary.apib http://127.0.0.1:3000

Authenticated APIs

Dredd supports all common authentication schemes:

  • Basic access authentication
  • Digest access authentication
  • OAuth (any version)
  • CSRF tokens

Use user setting in your configuration file or the :option:`--user` option to provide HTTP basic authentication:

--user=user:password

Most of the authentication schemes use HTTP header for carrying the authentication data. If you don’t want to add authentication HTTP header to every request in the API description, you can instruct Dredd to do it for you by the :option:`--header` option:

--header="Authorization: Basic YmVuOnBhc3M="

Sending Multipart Requests

.. literalinclude:: ../packages/dredd/test/fixtures/request/multipart-form-data.apib
  :language: apiblueprint

.. literalinclude:: ../packages/dredd/test/fixtures/request/multipart-form-data.yaml
  :language: openapi2

Sending Form Data

.. literalinclude:: ../packages/dredd/test/fixtures/request/application-x-www-form-urlencoded.apib
  :language: apiblueprint

.. literalinclude:: ../packages/dredd/test/fixtures/request/application-x-www-form-urlencoded.yaml
  :language: openapi2

Working with Images and other Binary Bodies

The API description formats generally do not provide a way to describe binary content. The easiest solution is to describe only the media type, to :ref:`leave out the body <empty-response-body>`, and to handle the rest using :ref:`hooks`.

Binary Request Body

API Blueprint

.. literalinclude:: ../packages/dredd/test/fixtures/request/image-png.apib
  :language: apiblueprint

OpenAPI 2

.. literalinclude:: ../packages/dredd/test/fixtures/request/image-png.yaml
  :language: openapi2

Hooks

In hooks, you can populate the request body with real binary data. The data must be in a form of a Base64-encoded string.

.. literalinclude:: ../packages/dredd/test/fixtures/request/image-png-hooks.js
  :language: javascript

Binary Response Body

API Blueprint

.. literalinclude:: ../packages/dredd/test/fixtures/response/binary.apib
  :language: apiblueprint

OpenAPI 2

.. literalinclude:: ../packages/dredd/test/fixtures/response/binary.yaml
  :language: openapi2

Note

Do not use the explicit binary or bytes formats with response bodies, as Dredd is not able to properly work with those (:ghissue:`api-elements.js#269`).

Hooks

In hooks, you can either assert the body:

.. literalinclude:: ../packages/dredd/test/fixtures/response/binary-assert-body-hooks.js
  :language: javascript

Or you can ignore it:

.. literalinclude:: ../packages/dredd/test/fixtures/response/binary-ignore-body-hooks.js
  :language: javascript

Multiple Requests and Responses

API Blueprint

To test multiple requests and responses within one action in Dredd, you need to cluster them into pairs:

FORMAT: 1A

# My API

## Resource [/resource/{id}]

+ Parameters
    + id: 42 (required)

###  Update Resource [PATCH]

+ Request (application/json)

        {"color": "yellow"}


+ Response 200 (application/json)

        {"color": "yellow", "id": 1}


+ Request Edge Case (application/json)

        {"weight": 1}

+ Response 400 (application/vnd.error+json)

        {"message": "Validation failed"}

Dredd will detect two HTTP transaction examples and will compile following transaction names:

$ dredd api-description.apib http://127.0.0.1 --names
info: Resource > Update Resource > Example 1
info: Resource > Update Resource > Example 2

In case you need to perform particular request with different URI parameters and standard inheritance of URI parameters isn’t working for you, try :ref:`modifying transaction before its execution <modifying-transaction-request-body-prior-to-execution>` in hooks.

OpenAPI 2

When using `OpenAPI 2`_ format, by default Dredd tests only responses with 2xx status codes. Responses with other codes are marked as skipped and can be activated in :ref:`hooks <hooks>`:

var hooks = require('hooks');

hooks.before('/resource > GET > 500 > application/json', function (transaction, done) {
  transaction.skip = false;
  done();
});

Using Apiary Reporter and Apiary Tests

Command-line output of complex HTTP responses and expectations can be hard to read. To tackle the problem, you can use Dredd to send test reports to `Apiary`_. Apiary provides a comfortable interface for browsing complex test reports:

$ dredd apiary.apib http://127.0.0.1 --reporter=apiary
warn: Apiary API Key or API Project Name were not provided. Configure Dredd to be able to save test reports alongside your Apiary API project: https://dredd.org/en/latest/how-to-guides/#using-apiary-reporter-and-apiary-tests
pass: DELETE /honey duration: 884ms
complete: 1 passing, 0 failing, 0 errors, 0 skipped, 1 total
complete: Tests took 1631ms
complete: See results in Apiary at: https://app.apiary.io/public/tests/run/74d20a82-55c5-49bb-aac9-a3a5a7450f06
Apiary Tests

Apiary Tests

Saving Test Reports under Your Account in Apiary

As you can see on the screenshot, the test reports are anonymous by default and will expire after some time. However, if you provide Apiary credentials, your test reports will appear on the Tests page of your API Project. This is great especially for introspection of test reports from Continuous Integration.

To get and setup credentials, just follow the tutorial in Apiary:

Apiary Tests Tutorial

Apiary Tests Tutorial

As you can see, the parameters go like this:

$ dredd -j apiaryApiKey:<Apiary API Key> -j apiaryApiName:<API Project Name>

In addition to using parameters and dredd.yml, you can also use environment variables:

  • APIARY_API_KEY=<Apiary API Key> - Alternative way to pass credentials to Apiary Reporter.
  • APIARY_API_NAME=<API Project Name> - Alternative way to pass credentials to Apiary Reporter.

When sending test reports to Apiary, Dredd inspects the environment where it was executed and sends some information about it alongside test results. Those are used mainly for detection whether the environment is Continuous Integration and also, they help you to identify individual test reports on the Tests page. You can use the following variables to tell Dredd what to send:

  • agent (string) - DREDD_AGENT or current user in the OS
  • hostname (string) - DREDD_HOSTNAME or hostname of the OS
  • CI (boolean) - looks for TRAVIS, CIRCLE, CI, DRONE, BUILD_ID, …

Example Values for Request Parameters

While example values are natural part of the API Blueprint format, the OpenAPI 2 specification allows them only for body request parameters (schema.example).

However, Dredd needs to know what values to use when testing described API, so it supports x-example :openapi2:`vendor extension property <vendorextensions>` to overcome the OpenAPI 2 limitation:

...
paths:
  /cars:
    get:
      parameters:
        - name: limit
          in: query
          type: number
          x-example: 42

The x-example property is respected for all kinds of request parameters except of body parameters, where native schema.example should be used.

Removing Sensitive Data from Test Reports

Sometimes your API sends back sensitive information you don’t want to get disclosed in :ref:`Apiary Tests <using-apiary-reporter-and-apiary-tests>` or in your CI log. In that case you can use :ref:`Hooks <hooks>` to do sanitation. Before diving into examples below, do not forget to consider following:

Sanitation of the Entire Request Body

Sanitation of the Entire Response Body

Sanitation of a Request Body Attribute

Sanitation of a Response Body Attribute

Sanitation of Plain Text Response Body by Pattern Matching

Sanitation of Request Headers

Sanitation of Response Headers

Sanitation of URI Parameters by Pattern Matching

Sanitation of Any Content by Pattern Matching

Sanitation of Test Data of Passing Transaction

Sanitation of Test Data When Transaction Is Marked as Failed in 'before' Hook

Sanitation of Test Data When Transaction Is Marked as Failed in 'after' Hook

Sanitation of Test Data When Transaction Is Marked as Skipped

Ultimate ‘afterEach’ Guard Using Pattern Matching

You can use this guard to make sure you won’t leak any sensitive data by mistake.

Sanitation of Test Data of Transaction With Secured Erroring Hooks

If your hooks crash, Dredd will send an error to reporters, alongside with current contents of the transaction.test (:ref:`docs <transaction-test>`) object. If you want to prevent this, you need to add try/catch to your hooks, sanitize the test object, and gracefully fail the transaction.