discussion: cucumber tests #95

Closed
emiltin opened this Issue Jan 29, 2012 · 12 comments

Projects

None yet

2 participants

@emiltin
Contributor
emiltin commented Jan 29, 2012

i've looked a bit more at the cucumber tests. the idea is to enable test based on small, well-known datasets. here's what tests look like:

    @routing @basic
    Feature: Basic Routing

    Scenario: Smallest possible datasat
        Given the nodes
         | a | b |

        And the ways
         | nodes |
         | ab    |

        When I route I should get
         | from | to | route |
         | a    | b  | ab    |
         | b    | a  | ba    |

    Scenario: Connected ways
        Given the nodes
         | a |   | c |
         |   | b |   |

        And the ways
         | nodes |
         | ab    |
         | bc    |

        When I route I should get
         | from | to | route |
         | a    | c  | abc   |
         | c    | a  | cba   |
         | a    | b  | ab    |
         | b    | a  | ba    |
         | b    | c  | bc    |
         | c    | b  | cb    |

    Scenario: Unconnected ways
        Given the nodes
         | a | b |
         | c | d |

        And the ways
         | nodes |
         | ab    |
         | cd    |

        When I route I should get
         | from | to | route |
         | a    | b  | ab    |
         | b    | a  | ba    |
         | c    | d  | cd    |
         | d    | c  | dc    |
         | a    | c  |       |
         | c    | a  |       |
         | b    | d  |       |
         | d    | c  |       |
         | a    | d  |       |
         | d    | a  |       |

    Scenario: Pick the fastest way type
        Given the nodes
         | a | s |
         | p | b |

        And the ways
         | nodes | highway   |
         | apb   | primary   |
         | asb   | secondary |

        When I route I should get
         | from | to | route |
         | a    | b  | apb   |
         | b    | a  | bpa   |

what goes on is:

  1. first you define a small node map. nodes are labelled with letters.
  2. then you define ways connecting the nodes. you can set any tags you want on ways.
  3. cucumber then writes he corresponding osm and speedprofile files, and processes them.
  4. the routing examples are them run one at a time, checking against expected behaviour.

here's an example run:

~/code/Project-OSRM$ cucumber --tags @basic
@routing @basic
Feature: Basic Routing

  Scenario: Smallest possible datasat # features/basic.feature:4
    Given the nodes                   # features/step_definitions/test_data.rb:124
      | a | b |
    And the ways                      # features/step_definitions/test_data.rb:139
      | nodes |
      | ab    |
    When I route I should get         # features/step_definitions/routing.rb:167
      | from | to | route                                                 |
      | a    | b  | ab                                                    |
      | b    | a  | ba                                                    |
      | a    | b  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | b    | a  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      Tables were not identical (Cucumber::Ast::Table::Different)
      ./features/step_definitions/routing.rb:180:in `/^I route I should get$/'
      features/basic.feature:12:in `When I route I should get'

  Scenario: Connected ways    # features/basic.feature:17
    Given the nodes           # features/step_definitions/test_data.rb:124
      | a |   | c |
      |   | b |   |
    And the ways              # features/step_definitions/test_data.rb:139
      | nodes |
      | ab    |
      | bc    |
    When I route I should get # features/step_definitions/routing.rb:167
      | from | to | route                                                 |
      | a    | c  | abc                                                   |
      | c    | a  | cba                                                   |
      | a    | c  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | a    | b  | ab                                                    |
      | c    | a  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | b    | a  | ba                                                    |
      | a    | b  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | b    | c  | bc                                                    |
      | b    | a  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | c    | b  | cb                                                    |
      | b    | c  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | c    | b  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      Tables were not identical (Cucumber::Ast::Table::Different)
      ./features/step_definitions/routing.rb:180:in `/^I route I should get$/'
      features/basic.feature:27:in `When I route I should get'

  Scenario: Unconnected ways  # features/basic.feature:36
    Given the nodes           # features/step_definitions/test_data.rb:124
      | a | b |
      | c | d |
    And the ways              # features/step_definitions/test_data.rb:139
      | nodes |
      | ab    |
      | cd    |
    When I route I should get # features/step_definitions/routing.rb:167
      | from | to | route                                                 |
      | a    | b  | ab                                                    |
      | b    | a  | ba                                                    |
      | a    | b  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | c    | d  | cd                                                    |
      | b    | a  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | d    | c  | dc                                                    |
      | c    | d  |                                                       |
      | a    | c  | (i) ""                                                |
      | d    | c  | (i) nil                                               |
      | c    | a  | (i) ""                                                |
      | a    | c  | (i) nil                                               |
      | b    | d  |                                                       |
      | c    | a  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | d    | c  | (i) ""                                                |
      | b    | d  | (i) nil                                               |
      | a    | d  | (i) ""                                                |
      | d    | c  | (i) nil                                               |
      | d    | a  | (i) ""                                                |
      | a    | d  | (i) nil                                               |
      | d    | a  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      Tables were not identical (Cucumber::Ast::Table::Different)
      ./features/step_definitions/routing.rb:180:in `/^I route I should get$/'
      features/basic.feature:46:in `When I route I should get'

  Scenario: Pick the fastest way type # features/basic.feature:59
    Given the nodes                   # features/step_definitions/test_data.rb:124
      | a | s |
      | p | b |
    And the ways                      # features/step_definitions/test_data.rb:139
      | nodes | highway   |
      | apb   | primary   |
      | asb   | secondary |
    When I route I should get         # features/step_definitions/routing.rb:167
      | from | to | route                                                     |
      | a    | b  | apb                                                       |
      | b    | a  | bpa                                                       |
      | a    | b  | Empty route: [["Head", "", 111, 0, 266, "0", "NE", 22.5]] |
      | b    | a  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]]     |
      Tables were not identical (Cucumber::Ast::Table::Different)
      ./features/step_definitions/routing.rb:180:in `/^I route I should get$/'
      features/basic.feature:69:in `When I route I should get'

Failing Scenarios:
cucumber features/basic.feature:4 # Scenario: Smallest possible datasat
cucumber features/basic.feature:17 # Scenario: Connected ways
cucumber features/basic.feature:36 # Scenario: Unconnected ways
cucumber features/basic.feature:59 # Scenario: Pick the fastest way type

4 scenarios (4 failed)
12 steps (4 failed, 8 passed)
1m56.321s

all tests are currently failing due to #88 (missing way names in routing instructions).

launching and quitting the binaries currently takes a while (perhaps due to large ram/disk allocations?) which makes the tests rather slow.

perhaps an option could be added to allocate small amounts when testing?

another idea would be to merge all three binaries into one, which could be kept running. it would then be instructed (perhaps by touching a file) to reprocess data. this would have the added benefit that request during reprocessing could be handled gracefully by sending back an appropriate status code.

Contributor
emiltin commented Jan 29, 2012

test for turn restrictions:

@routing @restrictions
Feature: Turn restrictions
    OSRM should handle turn restrictions as defined by http://wiki.openstreetmap.org/wiki/Relation:restriction

    Scenario: No left turn at T-junction
        Given the nodes
         | a | j | b |
         |   | s |   |

        And the ways
         | nodes |
         | aj    |
         | jb    |
         | sj    |

        And the relations
         | from | to | via | restriction  |
         | sj   | ja | j   | no_left_turn |

        When I route I should get
         | from | to | route |
         | a    | b  | ajb   |
         | a    | s  | ajs   |
         | b    | a  | bja   |
         | b    | s  | bjs   |
         | s    | a  |       |
         | s    | b  | sjb   |
Contributor
emiltin commented Jan 29, 2012

test for oneways

@routing @oneways
Feature: Oneway streets
    Handle oneways streets, as defined at http://wiki.openstreetmap.org/wiki/OSM_tags_for_routing

    Scenario: Implied oneways
        Then routability should be
         | highway       | junction   | forw | backw |
         | motorway      |            | x    | x     |
         | motorway_link |            | x    |       |
         | trunk         |            | x    | x     |
         | trunk_link    |            | x    |       |
         | primary       | roundabout | x    |       |

    Scenario: Overriding implied oneways
        Then routability should be
         | highway       | junction   | oneway | forw | backw |
         | motorway_link |            | no     | x    | x     |
         | trunk_link    |            | no     | x    | x     |
         | primary       | roundabout | no     | x    | x     |
         | motorway_link |            | -1     |      | x     |
         | trunk_link    |            | -1     |      | x     |
         | primary       | roundabout | -1     |      | x     |

    Scenario: Handle various oneway tag values
        Then routability should be
         | highway       | oneway   | forw | backw |
         | primary       |          | x    | x     |
         | primary       | nonsense | x    | x     |
         | primary       | no       | x    | x     |
         | primary       | false    | x    | x     |
         | primary       | 0        | x    | x     |
         | primary       | yes      | x    |       |
         | primary       | true     | x    |       |
         | primary       | 1        | x    |       |
         | primary       | -1       |      | x     |

    Scenario: Disabling oneways in speedprofile
        Given the speedprofile
         | obeyOneways | no |

    Then routability should be
         | highway       | junction   | oneway | forw | backw |
         | primary       |            | yes    | x    | x     |
         | primary       |            | true   | x    | x     |
         | primary       |            | 1      | x    | x     |
         | primary       |            | -1     | x    | x     |
         | motorway_link |            |        | x    | x     |
         | trunk_link    |            |        | x    | x     |
         | primary       | roundabout |        | x    | x     |

    Scenario: Oneways and bicycles 
        Given the speedprofile
         | accessTag   | bicycle |

        Then routability should be
         | highway       | junction   | oneway | oneway:bicycle | forw | backw |
         | primary       |            |        | yes            | x    |       |
         | primary       |            | yes    | yes            | x    |       |
         | primary       |            | no     | yes            | x    |       |
         | primary       |            | -1     | yes            | x    |       |
         | motorway      |            |        | yes            | x    |       |
         | motorway_link |            |        | yes            | x    |       |
         | primary       | roundabout |        | yes            | x    |       |
         | primary       |            |        | no             | x    | x     |
         | primary       |            | yes    | no             | x    | x     |
         | primary       |            | no     | no             | x    | x     |
         | primary       |            | -1     | no             | x    | x     |
         | motorway      |            |        | no             | x    | x     |
         | motorway_link |            |        | no             | x    | x     |
         | primary       | roundabout |        | no             | x    | x     |
         | primary       |            |        | -1             |      | x     |
         | primary       |            | yes    | -1             |      | x     |
         | primary       |            | no     | -1             |      | x     |
         | primary       |            | -1     | -1             |      | x     |
         | motorway      |            |        | -1             |      | x     |
         | motorway_link |            |        | -1             |      | x     |
         | primary       | roundabout |        | -1             |      | x     |

    Scenario: Cars should not be affected by bicycle tags
        Given the speedprofile
         | accessTag   | motorcar |

        Then routability should be
         | highway | junction   | oneway | oneway:bicycle | forw | backw |
         | primary |            | yes    | yes            | x    |       |
         | primary |            | yes    | no             | x    |       |
         | primary |            | yes    | -1             | x    |       |
         | primary |            | no     | yes            | x    | x     |
         | primary |            | no     | no             | x    | x     |
         | primary |            | no     | -1             | x    | x     |
         | primary |            | -1     | yes            |      | x     |
         | primary |            | -1     | no             |      | x     |
         | primary |            | -1     | -1             |      | x     |
         | primary | roundabout |        | yes            | x    |       |
         | primary | roundabout |        | no             | x    |       |
         | primary | roundabout |        | -1             | x    |       |
Contributor
emiltin commented Jan 29, 2012

it would be very useful for testing with an option to return the ID of all the intersection nodes passed by the route.

Owner

OSM IDs are discarded. This is a design choice because otherwise it bloats the memory consumption beyond any reasonable barrier.

Contributor
emiltin commented Jan 31, 2012

right. makes sense. i guess route instructions should identify the route without ambiguity

Owner

Emil, would you be interested in documenting cucumber testing on the wiki including the necessary setup and also a self-contained pull request?

Owner

To run these tests automatically (perhaps nightly) it would be great to have cucumber generate HTML output that can be automatically pushed to the projects web site. Also, sending mails when the nightly test breaks would be awesome.

Contributor
emiltin commented Feb 1, 2012

i could definately document this on the wiki. i'm glad you think this could be useful.
but i will need to work a bit more on it before it's ready.

for sure cucumber can work with a continuous integration setup. it has various hooks etc.

Contributor
emiltin commented Feb 1, 2012

i was hoping to specify expected routes by naming nodes, but since osrm doesn't include this directly i need to think about wether nodes can be implied from the route instructions, or expected routes should instead be specified by a series of ways.

Owner

I'd suggest to use a series of ways.

Contributor
emiltin commented Feb 1, 2012

yes that seems easiest. just wondering if it's always unambiguous? i guess it is, otherwise the routing instructions would be ambiguous.

Owner

One may artificially construct such a network, but I don't think this is an issue on a well-defined test instance.

@DennisOSRM DennisOSRM closed this Feb 14, 2012
@Zhdanovich Zhdanovich pushed a commit to Zhdanovich/Project-OSRM that referenced this issue Jul 20, 2014
DennisOSRM Adds the cucumber test framework referenced in issues #26, #95, #114 f91f17a
@Zhdanovich Zhdanovich pushed a commit to Zhdanovich/Project-OSRM that referenced this issue Jul 20, 2014
@emiltin emiltin + DennisOSRM Adds the cucumber test framework referenced in issues #26, #95, #114 ced10e2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment