Cucumber Test Suite
Clone this wiki locally
The correct functionality of OSRM is supported by a test suite based on Cucumber (https://cucumber.io/). The tests are run automatically on Travis CI (https://travis-ci.org/Project-OSRM/osrm-backend).
- Running tests locally
- Tests explained
- Writing tests
- Customizing Tests
The test suite currently does not have complete coverage for all osrm-backend functionality, but we are striving for full test coverage.
Running tests locally
Whenever you are making a change to osrm-backend, you should run the test suite locally to verify that the change does not introduce regressions. If your change introduces new or changed behaviour, you need to adjust the test suite accordingly.
osrm-backendbuilt locally as explained here
- Install Node.js
- Tests require Node 4.0+. The
nvmproject is very helpful for installing and switching between Node versions.
- Tests require Node 4.0+. The
- Install Node dependencies
- Once Node.js 4.0+ is installed you can install modules via
npm installwill install all dependencies listed in
package.json, then run
npm linkto link
cucumberas an executable.
- Once Node.js 4.0+ is installed you can install modules via
The following profiles each run different subsets of cucumber tests.
cucumber # default profile, runs all tests that should pass (exclude @todo and @bug) cucumber -p verify # same as above, but with minimal output format cucumber -p bugs # run all tests covering known bugs (@bug) cucumber -p todo # run all tests covering unimplemented features (@todo) cucumber -p all # run all tests, including @todo and @bug
You can also run specific tests or groups of tests:
- To run tests with a specific tag, use:
cucumber --tags @restrictions
- To exclude tests withs a specific tag, use:
cucumber --tags ~@restrictions
- To run a specific scenario:
- To run a named scenario:
cucumber --name "A single way with two nodes"
- To run all scenario in a specific folder:
Note that the default profile excludes tests tagged with @todo and @bug, so if you want to run a test with one of those tags, you need to use the 'all' profile:
cucumber -p all -t @weird_bug_7
Tests are evaluated by parsing the output returned by
osrm-routed. Results are shown in the terminal.
A passed route will be shown using a green row. A failed row will be shown with two rows: The expected result in a yellow row, followed by the actual result in a grey row.
(Note that there's currently an open cucumber-js issue about implementing formatted diffs)
During the tests, several log files are used:
- Output from
osrm-contractare redirected to one file located at
- Output from
osrm-routedis redirected to
The general structure of executing one cucumber test scenario is this:
- Create an OSM fixture file
- Run the OSM fixture file through
osrm-routedwith the generated dataset
- Make a request against
- Compare request results with expectations
Let's dive into details:
Tests are located in a sub-directory called
features/ and organized into feature files, each containing many scenarios.
Actual test runs are performed inside the folder
test/. The folder contain a .stxxl config file using a small 10MB disk.
For each test scenario, an
.osm data file is constructed based on the description in the test scenario.
It's then preprocessed with
osrm-contract. The relevant profile script is copied to
test/profile.ini before each preprocessing, so the binaries pick it up.
.osrm files are cached, so that preprocessing is only repeated if either the
.osm data, the profile or the binaries have changed. This helps to speed up re-running of tests.
test/server.ini is now rewritten to ensure that the right configuration is used.
osrm-routed is then launched, and routes are requested via http calls to
localhost:5000, and the response is compared to the expected result. All request are timed out after 10 seconds. Both timeouts and crashes are caught and reported.
To ensure a clean run of every test, all processes named
osrm-routed are killed before and after each test, using a signal 9, no matter who launched them.
For Cucumber documentation, see https://github.com/cucumber/cucumber/wiki/A-Table-Of-Content.
The OSRM tests currently come in three main flavors:
With node maps you geometrically place nodes on a mini map. Each node is identified by a letter. You then string together nodes to create ways, and optionally add relations. Finally you specify a list of routes to be tested, described by start and end points, and the expected route:
Scenario: Zig Zag Given the node map """ a c \ / b """ And the ways | nodes | | ab | | bc | When I route I should get | from | to | route | | a | c | ab,bc,bc | | c | a | bc,ab,ab |
The final segment of the route appears twice in this response. This is actually desired behaviour. The first part is for a turn onto the segment, the second one is for the arrival operation. The full list of instructions would be
arrive. Both the
new name and
arrive indicate the same segment, hence the replication of the segment in
Under 'Given the node map', the node map is constructed using nodes. Each node must be specified by a letter
a-z. Only one node can exists at each location, and a node can only be present once on the map. Other characters like \ / | - can be used to visually indicate connections,but note that these characters are in fact ignored by the test. To construct ways you need to use an "And the ways" table.
When we generate the
.osm file for the test case from the node map, node IDs are assigned, starting at 1, in left-to-right, top-to-bottom order. In the example above, a=1, c=2, b=3.
You can add OSM tags to nodes by adding using a
And the nodes table:
And the nodes | node | barrier | | a | bollard | | b | gate |
node column identifies the node, while additional columns are added as tags to the corresponding node.
In addition to the nodes specified with
a-z, you can add locations specified with numbers
0-9. Locations can be used as start/end/via points when requesting routes, but are not included in the OSM file, meaning you add/move them without affecting the OSM input data.
For example, this map results in the same OSM input as the example above, but you can test a route from 1 to 2.
Given the node map | a | | c | | 1 | b | 2 |
And the ways, you create ways by stringing together nodes in the node map. For example,
ab is a way starting at a and ending at b, while
abc is a way starting at a, passing b, and ending at c. A ways can have two or more nodes. All nodes of the way must exist in the node map.
You can add OSM tags to ways by adding columns to the
And the ways table:
And the ways | nodes | oneway | | ab | yes | | bc | no |
Given ... and
And ... is interchangeable in cucumber, and you should just use what creates a natural language flow.
Specifying routes to test
An OSM file is constructed based on the given nodes, ways and relations, and then used for testing. Using
When I route I should get you specify routes to be tested, each with a start and an end node, and the ways you expect the route to follow:
When I route I should get | from | to | route | | a | c | ab,bc,bc | | c | a | bc,ab,ab |
An alternative syntax uses a single
waypoints column to specify the nodes you want to follow:
When I route I should get | waypoints | route | | a,c | ab,bc,bc | # from a, to b | a,b,c | ab,bc,bc | # from a, via b, to c
If you specify more than two waypoints, via points are used.
Note that you cannot specify the nodes you expect the computed route to follow, only the ways. This is because OSRM does not return nodes in query results, only ways.
Testing Distance and Travel Time
In addition to the route itself, you can test for total length or time (or both) by adding the corresponding columns to the test table:
When I route I should get | from | to | route | time | distance | | a | b | ab,ab | 24s | 100m | | c | d | cd,cd | 72s | 200m |
Time is specified in seconds, while length is specified in meters. (Note that OSRM currently quantized length to the nearest 10m, and time to the nearest second)
OSRM uses spherical coordinates. Distances in node maps deviate from the naive flat geometry when you move away from the equator.
Testing Turn Instructions
You can test the turn instructions by adding a "turns" columns to the test table:
When I route I should get | from | to | route | turns | | a | c | ab,bc,bc | depart,left,arrive | | c | a | bc,ab,ab | depart,right,arrive |
The expected turns are specified using words that correspond to the v5 turns and modiffiers that are returned by OSRM:
depart arrive new name [uturn|sharp right|right|slight right|straight|slight left|left|sharp left] continue [uturn|sharp right|right|slight right|straight|slight left|left|sharp left] turn [uturn|sharp right|right|slight right|straight|slight left|left|sharp left] merge [uturn|sharp right|right|slight right|straight|slight left|left|sharp left] ramp [uturn|sharp right|right|slight right|straight|slight left|left|sharp left] fork [uturn|sharp right|right|slight right|straight|slight left|left|sharp left] end of road [uturn|sharp right|right|slight right|straight|slight left|left|sharp left] roundabout-exit-[#|undefined] rotary-exit-[#|undefined] roundabout turn [uturn|sharp right|right|slight right|straight|slight left|left|sharp left] notification
Fuzzy ranges for expected numerical values can be specified using either a percentage range, or an absolute +- range:
When I route I should get | from | to | route | distance | | a | d | abcde,abcde | 300m +-1 | | a | e | abcde,abcde | 700m ~1% |
You can add a comment column by using the column name
#. The column will be ignored, but can be useful if you need to explain individual rows:
When I route I should get | from | to | route | time | # | | a | d | abcde,abcde | 30s | we're going downhill | | a | e | abcde,abcde | 60s | we're going uphill |
Lat/lon tables are quite similar to node maps, except the nodes are placed using a table with absolute latitude and longitude values:
Scenario: Longitudinal distances at latitude 45 Given the node locations | node | lat | lon | | c | 45 | 80 | | d | 45 | 0 | And the ways | nodes | | cd | When I route I should get | from | to | route | distance | | c | d | cd,cd | 6028844 ~0.5% |
This is especially useful for testing things related to the Mercator projection and huge distances. In this case, node IDs are assigned in the
.osm file in top-to-bottom order. In the example above, c=1, d=2.
Response codes of OSRM queries can be checked using a
code column as
When I route I should get | from | to | code | | a | b | NoRoute |
If annotations are included in the request,
Given the query options | annotations | true |
then they can also be checked for in the response output with the following syntax:
When I route I should get | from | to | route | a:nodes | a:duration | | a | c | ab,bc,bc | 1:2,2:3 | 10,2.5 |
The fields that can be checked for are the fields that can be requested with the annotations parameter:
duration, distance, datasources, nodes, weight.
: delineates annotation values within an annotation object within a step, a
, delineates grouped data from the same leg object.
This flavor of tests is used for checking individual isolated ways, and is often useful for testing access and barrier tags. You can test each direction (forward/backward) separately:
Scenario: Simplest possible oneway Given the speedprofile "car" Then routability should be | highway | oneway | forw | backw | | primary | | x | x | | primary | yes | x | | | primary | -1 | | x |
Column headers are used to specify tag keys. In the example above, the
oneway tags are used.
You can also set tags on a central node of the way, by using columns with titles in the form of
Scenario: Car - Barriers Then routability should be | highway | node/barrier | bothw | | primary | | x | | primary | bollard | |
bothw have special meaning:
- If a
forw_column is present, the way is tested for forward routability
- If a
backw_column is present, the way is tested for backwards routability
- If a
_bothw_column is present, the way is tested for both forwards and backwards routability
A single OSM file is constructed, containing an isolated way for each row. The ways are labelled
w3, etc and placed in a line, but with spacing in between:
--w1-- --w2-- --w3--
For all types of tests, nodes are by default spaced on a grid with origin at
1,1 and a grid size of
100m. The grid size can be changed using:
Given a grid size of 500 meters
The origin can be changed by:
Given the origin 55.5,12.2
Note that grid spacing uses fixed lat/lon increments (ignoring the latitude), and will therefore be distorted when you move away from the equator. For this reason, nodes far ways from the equator are often better placed using absolute node locations.
By default, tests use the "bicycle" profile in
profiles/bicycle.lua, but this can be changed for each scenario by adding:
Given the speedprofile "car"
Note that you can use a
Background: section if you need to specify things for all test scenarios in a
Background: Given the speedprofile "car" Given a grid size of 500 meters