Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Cucumber Test Suite

Emil Tin edited this page · 31 revisions

The correct functionality of OSRM is supported by a test suite based on Cucumber (http://cukes.info). If you don't want to run tests, you can build and run OSRM without doing any of the following.

Cucumber itself is written in Ruby, and is distributed as a Ruby gem. All tests are written in Ruby using the natural language Gherkin syntax.

Tests are run automatically on http://ci.osm.org whenever commits are pushed to the GitHub repo.

Prerequisites

  • osmosis
  • ruby 1.9+: Some system come with 1.8, so you might need to update.
  • gems: cucumber, rake, osmlib-base, sys-proctable, rspec-expectations. The easiest is to first install the bundler gem using gem install bundler, then run bundle install from the project folder. This will automatically install the needed gems.
vi /var/lib/gems/1.8/specifications/term-ansicolor-1.0.7.gemspec

and change the s.date = line to the following: s.date = %q{2011-10-13} and save it.

Files

Tests are located in a sub-directory called 'features/' and organized into feature files, each containing scenarios.

Actual test runs are performed inside the folder 'test/'. The folder contain a .stxxl config file using a small 10MB disk.

Mechanics

For each test scenario, an .osm data file is constructed based on the description in the test scenario. The .osm file is then converted to .pbf format using the osmosis command line tool.

It's then preprocessed with osrm-extract and osrm-prepare. The relevant profile script is copied to 'test/profile.ini' before each preprocessing, so the binaries pick it up.

The resulting .osrm files are cached, so that preprocessing is only repeated if either the osm data, the profile or the binaries have changed. This helps to speed rerunning of tests.

'test/server.ini' is now rewritten to ensure that the right configuration is used. osrm-routed is then launched, and routes are requested via http calls to localhost:5000, and the response is compared to the expected result. All request are timed out after 10 seconds. both timeouts and crashes are caught and reported. To ensure a clean run of every test, all processes named osrm-extract, osrm-prepare and osrm-routed are killed before and after each test, using a signal 9, no matter who launched them.

Writing tests

For Cucumber documentation, see https://github.com/cucumber/cucumber/wiki/A-Table-Of-Content.

The OSRM tests currently come in three main flavors: node maps, lat/lon tests, and routability tables.

Node Maps

With node maps you geometrically place nodes on a mini map. Each node is identified by a letter. You then string together nodes to create way, and optionally add relations. Finally you specify a list of routes to be tested, described by start and end points, and the expected route:

Scenario: Zig Zag
    Given the node map
     | a |   | c |
     |   | b |   |

    And the ways
     | nodes |
     | ab    |
     | bc    |

    When I route I should get
     | from | to | route |
     | a    | c  | ab,bc |
     | c    | a  | bc,ab |

Nodes

Under 'Given the node map', the node map is constructed using the pipe character '|' to space nodes. The node map can have any size you want, but must be rectangular. Each node must be specified by a letter a-z. Only one node can exists at each location, and a node can only be present once on the map.

You can add OSM tags to nodes by adding using a 'And the nodes' table:

    And the nodes
     | node | barrier |
     | a    | bollard |
     | b    | gate    |

The 'node' column identifies the node, while additional columns are added as tags to the corresponding node.

Locations

In addition to the nodes specified with a-z, you can add locations specified with numbers 0-9. Locations can be used as start/end/via points when requesting routes, but are not included in the OSM file, meaning you add/move them without affecting the OSM input data.

For example, this map results in the same OSM input as the example above, but you can test a route from 1 to 2.

    Given the node map
     | a |   | c |
     | 1 | b | 2 |

Specifying ways

Under 'And the ways', you create ways by stringing together nodes in the node map. For example, 'ab' is a way starting at a and ending at b, while 'abc' is a way starting at a, passing b, and ending at c. A ways can have two or more nodes. All nodes of the way must exist in the node map.

You can add OSM tags to ways by adding columns to the 'And the ways' table:

    And the ways
     | nodes | oneway |
     | ab    | yes    |
     | bc    | no     |

Note that 'Given...' and 'And...' is interchangeable in cucumber, and you should just use what creates a natural language flow.

Specifying routes to test

An OSM file is constructed based on the given nodes, ways and relations, and then used for testing. Using 'When I route I should get' you specify routes to be tested, each with a start and an end node, and the ways you expect the route to follow:

    When I route I should get
     | from | to | route |
     | a    | c  | ab,bc |
     | c    | a  | bc,ab |

An alternative syntax uses a single 'waypoints' column to specify the nodes you want to follow:

    When I route I should get
     | waypoints | route |
     | a,c       | ab,bc | # from a, to b
     | a,b,c     | ab,bc | # from a, via b, to c

If you specify more than two waypoints, via points are used.

Note that you cannot specify the nodes you expect the computed route to follow, only the ways. This is because OSRM does not return nodes in query results, only ways. For each row, osrm-routed is queried on port 5000, and the result parsed and compared to the expected route.

Testing Distance and Travel Time

In addition to the route itself, you can test for total length or time (or both) by adding the corresponding columns to the test table:

    When I route I should get
     | from | to | route | time | distance |
     | a    | b  | ab    | 24s  | 100m     |
     | c    | d  | cd    | 72s  | 200m     |

Time is specified in seconds, while length is specified in meters. (Note that OSRM currently quantized length to the nearest 10m, and time to the nearest second.)

OSRM uses spherical coordinates. Distances in node maps deviate from the naive flat geometry when you move away from the equator.

Testing Turn Instructions

You can test the turn instructions by adding a "turns" columns to the test table:

    When I route I should get
     | from | to | route | turns                  |
     | a    | c  | ab,bc | head,left,destination  |
     | c    | a  | bc,ab | head,right,destination |

The expected turns are specified using words that correspond to the integer codes that are returned by OSRM:

0: none 1: straight 2: slight_right 3: right 4: sharp_right 5: u_turn 6: sharp_left 7: left 8: slight_left 9: via 10: head 11: enter_roundabout 12: leave_roundabout 13: stay_roundabout 14: start_end_of_street 15: destination 16: enter_contraflow 17: leave_contraflow

Enter_roundabout is special, since OSRM returns a instruction string in the form of "11-2", where 11 is the actual instruction, and 2 is the exit nr. Use a string in the in form "enter_roundabout-2" to test this. OSRM doesn't currently use leave_roundabout.

Fuzzy ranges

Fuzzy ranges for expected numerical values can be specified using either a percentage range, or an absolute +- range:

    When I route I should get
     | from | to | route | distance |
     | a    | d  | abcde | 300m +-1 |
     | a    | e  | abcde | 700m ~1% |

Comment column

You can add a comment column by using the column name "#". The column will be ignored, but can be useful if you need to explain individual rows:

    When I route I should get
     | from | to | route | time | \#                    |
     | a    | d  | abcde | 30s  | we're going downhill |
     | a    | e  | abcde | 60s  | we're going uphill   |

Lat/lon Tables

Lat/lon tables are quite similar to node maps, except the nodes are placed using a table with absolute latitude and longitude values:

    Scenario: Longitudinal distances at latitude 45
        Given the node locations
         | node | lat | lon |
         | c    | 45   | 80 |
         | d    | 45   | 0  |

        And the ways
         | nodes |
         | cd    |

        When I route I should get
         | from | to | route | distance      |
         | c    | d  | cd    | 6028844 ~0.5% |

This is especially useful for testing things related to the Mercator projection and huge distances.

Routability Tables

This flavor of tests is used for checking individual isolated ways, and is often useful for testing access and barrier tags. You can test each direction (forward/backward) separately:

Scenario: Simplest possible oneway
    Given the speedprofile "car"
    Then routability should be
     | highway | oneway | forw | backw |
     | primary |        | x    | x     |
     | primary | yes    | x    |       |
     | primary | -1     |      | x     |

Column headers are used to specify tag keys. In the example above, the 'highway' and 'oneway' tags are used.

You can also set tags on a central node of the way, by using columns with titles in the form of 'node/...':

    Scenario: Car - Barriers 
        Then routability should be
         | highway | node/barrier   | bothw |
         | primary |                | x     |
         | primary | bollard        |       |

Columns forw, backw and bothw has special meaning. If an forw column is present the way is tested for forward routability. If an backw row is present, it's tested for backwards routability. If an bothw row is present, it's tested for both forwards and backwards routability.

A single OSM file is constructed, containing an isolated way for each row. The ways are labelled w1, w2, w3, etc and placed in a line, but with spacing in between:

--w1--    --w2--    --w3--    

Customizing Tests

For both types of tests, nodes are by default spaced on grid with origin at 1,1 and a grid size of 100m. The grid size can be changed using:

    Given a grid size of 500 meters

The origin can be changed by:

    Given the origin 55.5,12.2

Note that grid spacing uses fixed lat/lon increments (ignoring the latitude), and will therefore be distorted when you move away from the equator. For this reason, nodes far ways from the equator are often better placed using absolute node locations.

By default, tests use the "bicycle" profile in profiles/bicycle.lua, but this can be changed for each scenario by adding:

    Given the speedprofile "car"

Note that you can use a 'Background:' section if you need to specify things for all test scenarios in a .feature file:

    Background:
        Given the speedprofile "car"
        Given a grid size of 500 meters

Running Tests

To verify that a change did not introduce regression errors, it's recommended to run all that should pass. A few profiles are setup for common workflows:

cucumber              # default profile, runs all tests that should pass (exclude @todo and @bug)
cucumber -p verify    # same as above, but with minimal output format
cucumber -p bugs      # run all tests covering known bugs (@bug)
cucumber -p todo      # run all tests covering unimplemented features (@todo)
cucumber -p all       # run all tests, including @todo and @bug

You can also run specific tests or groups of tests:

  • To run tests with a specific tag, use: cucumber --tags @restrictions
  • To exclude tests withs a specific tag, use: cucumber --tags ~@restrictions
  • To run a specific scenario: cucumber features/bicycle/restrictions.feature:6
  • To run a named scenario: cucumber --name "A single way with two nodes"
  • To run all scenario in a specific folder: cucumber features/foot

Note that the default profile excludes tests tagged with @todo and @bug, so if you want to run a test with one of those tags, you need to use the 'all' profile:

cucumber -p all -t @weird_bug_7

Output

Tests are evaluated by parsing the instructions hash returned by osrm-routed. Results are shown in the terminal. A passed routed will be shown using a green row. A failed row will be shown with two rows: The expected result in a yellow row, followed by the actual result in a grey row.

(Note that there's currently an cucumber issue, causing rows below the first failed row to incorrectly by marked as failed.)

Log files

During the tests, several log files are used. Output from osrm-extract and osrm-prepare are redirected to test/preprocess.log. Output from osrm-routed is redirected to test/osrm-routed.log.

Additional info about failed tests is logged to fail.log, containing details about the osm data used, the profile, expected and returned routes, as well as actual query and response involved.

Something went wrong with that request. Please try again.