Data-driven testing of FBP components and graphs
jonnor Merge pull request #156 from flowbased/greenkeeper/commander-2.16.0
fix(package): update commander to version 2.16.0
Latest commit ea410b0 Jul 16, 2018
Permalink
Failed to load latest commit information.
bin Distribute fbp-spec as JavaScript instead of CoffeeScript Nov 7, 2017
browser Add CNAME Apr 22, 2016
doc doc: Update braindump Dec 7, 2015
examples schema: Update to allow sequences of inputs/expect Apr 20, 2016
protocol-examples @ d014f12 Check capabilities sent by runtime before running suites, fixes #118 Mar 23, 2018
schema schema: Avoid dynamic require() calls Feb 19, 2017
schemata schema: Update to allow sequences of inputs/expect Apr 20, 2016
spec Switch to a more expansive ES6 polyfill for PhantomJS Mar 22, 2018
src Allow to specify FBP command timeout, bump default Jul 16, 2018
ui ui: Get React from NPM Feb 19, 2017
.gitignore Ignore package-lock.json Mar 22, 2018
.gitmodules Tests: Add check mocha.run() using fbp-python-example runtime Apr 21, 2015
.npmignore Distribute fbp-spec as JavaScript instead of CoffeeScript Nov 7, 2017
.travis.yml Travis: Try update NPM token again Jul 16, 2018
CHANGES.md Release 0.6.2 Mar 30, 2018
Gruntfile.coffee Scope build also to target Mar 23, 2018
LICENSE.md Release 0.6.0 Mar 30, 2018
README.md Update status Nov 17, 2017
browser.js ui: Only include for browser build Feb 19, 2017
index.js Distribute fbp-spec as JavaScript instead of CoffeeScript Nov 7, 2017
package.json Merge pull request #156 from flowbased/greenkeeper/commander-2.16.0 Jul 16, 2018
webpack.config.js Babelize ES6 code for the browser build Mar 22, 2018
webpack.entry.js Make require work in webpack'd bundle Nov 6, 2017

README.md

fbp-spec Build Status Greenkeeper badge

A runtime-independent test framework for Flow Based Programming (FBP) component and graphs, using declarative, data-driven testing.

One can use fbp-spec to do testing at multiple levels, each approximately corresponding to the different architectural levels of Flow Based Programming:

  • Unit (FBP component/subgraph)
  • Integration (FBP graph)
  • System (FBP runtime)

Status:

In production

Purpose & Scope

Note: fbp-spec is intended for use by application and component-library developers.

The following is considered out-of-scope:

  • Testing conformance with the FBP protocol. Instead use fbp-protocol
  • Testing an FBP runtime/engine itself. Instead use a testing framework for your particular runtime language/environment.

License

The MIT license

Usage

Installing

Set up fbp-spec as an NPM dependency

npm install --save-dev fbp-spec

or, install it globally. Useful if you just want the commandline tool.

npm install -g fbp-spec

Writing tests

Each declared test suite loads an FBP component (or graph) fixture, and runs a set of test cases by sending a set of input data to input ports and verifying the output data against the expected results.

name: "Simple example of passing tests"
topic: "core/Repeat"
fixture:
 type: 'fbp'
 data: |
  INPORT=it.IN:IN
  OUTPORT=f.OUT:OUT
  it(core/Repeat) OUT -> IN f(core/Repeat)

cases:
-
  name: 'sending a boolean'
  assertion: 'should repeat the same'
  inputs:
    in: true
  expect:
    out:
      equals: true
-
  name: 'sending a number'
  assertion: 'should repeat the same'
  inputs:
    in: 1000
  expect:
    out:
      equals: 1000

Multiple ports

You can send data to multiple inports and check expectations on multiple ports per testcase:

-
  name: '1 active track toggled high'
  assertion: 'should give value1 color'
  inputs:
    tracks: 1
    animation: [
      0, # track idx
      "0xEE00EE", # val0
      "0xAA00AA", # val1
      200, # period
      50, # dutycycle
      0, # offset
      500 ] # duration
    clock: 250
  expect:
    clock:
     equals: 250
    value:
     equals: [0, 0x00AA] # FIXME: truncated

Sequence of packets

For testing components with state, you can sending multiple input packets in sequence.

-
  name: 'sequence of data using spacy notation'
  assertion: 'should pass'
  inputs:
    -
      in: true
    -
      in: false
  expect:
    - 
      out:
        equals: true
    -
      out:
        equals: false 

Extract data using path

With path you can specify a JSONPath to extract the piece(s) of data the assertions will be ran against:

-
  name: 'select single value'
  assertion: 'should pass'
  inputs:
    in: { outer: { inner: { foo: 'bar' } } }
  expect:
    out:
      path: '$.outer.inner.foo'
      equals: 'bar'
-
  name: 'selecting many correct values'
  assertion: 'should pass'
  inputs:
    in:
      outer:
        first: { foo: 'bar' }
        second: { foo: 'bar' }
  expect:
    out:
      path: '$.outer.*.foo'
      equals: 'bar'

Skipping tests

Setting skip property on a testcase or suite, will cause it to not be ran. Should contain a message of the reason for skipping.

-
  name: 'a test that is skipped'
  assertion: 'will not be ran'
  inputs:
    in: 1000
  expect:
    out:
      equals: 1000
  skip: 'not implemented yet'

Using fixtures

One can use testing-specific components in the fixture, to simplify driving the unit under test with complex inputs and performing complex assertions.

fixture:
 type: 'fbp'
 data: |
  INPORT=imagename.IN:NAME
  INPORT=testee.PARAM:PARAM
  INPORT=reference.IN:REFERENCE
  OUTPORT=compare.OUT:SIMILARITY

  generate(test/GenerateTestImage) OUT -> IN testee(my/Component)
  testee OUT -> ACTUAL compare(test/CompareImage)
  reference(test/ReadReferenceImage) OUT -> REFERENCE compare
cases:
-
  name: 'testing complex data with custom components fixture'
  assertion: 'should pass'
  inputs:
    name: someimage
    param: 100
    reference: someimage-100-result
  expect:
    similarity:
      above: 0.99

Supported assertions

Instead of equals you can use any of the supported assertion predicates. Examples include:

type
above
below
contains
haveKeys
includeKeys

For a full set of assertions, see the schema

More

A comprehensive set of examples can be found under ./examples. For the detailed definition of the dataformat for tests, see schemata/.

Running tests with fbp-spec commandline tool

The simplest and most universal way of running tests is with the fbp-spec commandline tool.

$ fbp-spec --address ws://localhost:3333 examples/multisuite-failandpass.yaml

MultiSuite, failing tests
  sending a boolean with wrong expect
    should fail: ✗ Error: expected true to deeply equal false
  sending a number with wrong expect
    should fail: ✗ Error: expected 1000 to deeply equal 1003
MultiSuite, passing tests
  sending a boolean
    should repeat the same: ✓
  sending a number
    should repeat the same: ✓

The --command options can be used to specify a command which will start the runtime under test:

fbp-spec --command "python2 protocol-examples/python/runtime.py"

It sets the exit status to non-zero on failure, so is suitable for integrating into a Makefile or similar.

Running tests by integrating with Mocha

Mocha iss a popular test runner framework for JavaScript/CoffeeScript on browser and node.js.

Since fbp-spec communicates with your runtime over a network protocol, you can use this also when your project is not JavaScript-based. The Mocha runner is for instance used in microflo-core to test C++ components for microcontrollers & embedded devices.

You can have your fbp-spec tests run in Mocha by calling the fbpspec.mocha.run() function, in a file which is executed with the standard Mocha runner. Eg. mocha --reporter spec tests/fbpspecs.js

// fbpspecs.js
fbpspec = require('fbp-spec');

rt = {
  protocol: "websocket",
  address: "ws://localhost:3569",
  secret: "py3k", // Optional. If needed to connect/authenticate to runtime
  command: 'python2 protocol-examples/python/runtime.py' // Optional. Can be used to start runtime automatically
};
fbpspec.mocha.run(rt, './examples/simple-passing.yaml', { starttimeout: 1000 });

The tests can be specified as a list of files, or directories. You can use the standard grep option of Mocha to run only some tests.

For CoffeScript example, see ./spec/mocha.coffee.

Running tests interactively in Flowhub

Flowhub IDE (version 0.11 and later) has integrated support for fbp-spec. No installation is required.

  • Open existing project, or create a new one
  • Open a component, and write/copypaste in a test in the Tests panel
  • Ensure you have a runtime set up, and connected

When you make changes to your project (components,graphs) or tests, Flowhub will now automatically (re-)run your tests. You can see the status in the top-right corner. Clicking on it brings up more details.

Generating tests programatically

The test-format defined by fbp-spec is fairly generic and versatile. It is intended primarily as a format one directly specifies tests in, but can also be generated from other sources.

Sometimes data-driven testing, one does a large amount of very similar tests, with multiple test-cases per set of input data. By capturing only the unique parts of testcases in a specialied data-structure (JSON, YAML, etc), and then transforming this into standard fbp-spec files with some code, adding/removing cases becomes even easier. For instance in imgflo-server, testcases can be defined by providing a name, an URL and a reference result (a file with naming convention based on name).

Similarly, one can generate testcases using fuzzing, schema-based, model-based or similar tools.

Integrating test runner in an application

The test runner code is accessible as a JavaScript library, and can be integrated into other apps (like Flowhub does). See examples of commandline and webappp usage.

Add supporting for a new runtime

You need to implement the FBP network protocol. At least the protocol:runtime, protocol:graph, and protocol:network capabilities are required.

All transports supported by fbp-protocol-client are supported by fbp-spec, including WebSocket, WebRTC, and iframe/postMessage.

fbp-spec is intended to be used with flow-based and dataflow-programming, but might be useful also outside these programming paradigms. Try it out!

Writing a test runner in another language

As long as you stay compatible with the fbp-spec testformat and FBP protocol, you can implement a compatible runner in any programming language.

You can consider the fbp-spec code (in CoffeeScript) as a reference implementation.