Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

cleaned up cucumber tests #114

Closed
wants to merge 5 commits into from

2 participants

@emiltin
Collaborator

here's a cleaned up version of cucumber tests - single commit rebased on your master.

it's quite a big chunk with many pieces, so take a look, and hopefully we can iron out any issues, so it can become a help in finding and fixing bugs.

Prerequisites:
ruby: should be preinstalled on most systems.
gems: cucumber, rake, osmlib-base. the easiest and safest is to first install the 'bundler' gem, then run 'bundle install' from the project folder.
osmosis: command line tool for converting and cropping osm files. if you're on mac you can install it via homebrew with 'brew install osmosis'

if you don't want to run tests, you should still be able to build and run osrm without installing any of the above.

SConstruct:
the SConstruct file had to rearranged a bit to get os x compilation to work. other platforms should not be affected, but i couldn't test it on those other platforms. compilation on mac does not currently use openmp.

Files:
test are located in features/ and organized into feature files, each containing scenarios.

test and sandbox are now two separate folders. test is where all the tests are run. sandbox is where you can use rake task to download real osm data, run the server manually to test in a browser, etc.

speedprofile.ini has been split into separate files, stored in speedprofiles/

both test/ and sandbox/ folder contains a .stxxl config file. the one in the test folder uses a 1MB disk, the one in sandbox uses a 1GB disk.

Mechanics:
for each test scenario, an osm data file is constructed based on the description in the test scenario. the osm file is converted to protobuffer and preprocessed with osrm-extract and osrm-prepare.

the resulting osrm files are cached, so that processing is only repeated if either the osm data, the speedprofile or the binaries have changed. this helps speed up the running of tests a lot.

test/server.ini and test/speedprofile.ini is now rewritten to ensure that the right data files are used.

osrm-routed is then launched, and routes are requested via http calls to port 5000 on localhost, and the response is compared to the expected result.

all request are timed out after 5 seconds. both timeouts and crashes are caught and reported.

to ensure a clean run of every test, all processes named osrm-extract, osrm-prepare and osrm-routed are killed before and after each test using a signal 9, no matter who launched them. i have only tested this on mac os x 10.7. it should work on linux, but process management can often behave differently depending on platform. test are not yet expected to work on windows, for example the unix shell command 'ps' is used. i assume there will be other issue too, but if course it should be possible to get it working.

Writing tests:
tests come in two main flavors. one where you specify data and test routes separately, and one where you just provide a table of ways and their tags to be tests.

with the first type, you setup the data by a small 'node map', and a list of ways (and optionally relations). finally you specify a list of start/end pairs, and the expected route, described by the ways traversed:

Scenario: Zig Zag
    Given the nodes
     | a |   | c |
     |   | b |   |

    And the ways
     | nodes |
     | ab    |
     | bc    |

    When I route I should get
     | from | to | route |
     | a    | c  | ab,bc |
     | c    | a  | bc,ab |

by default, tests use the bicycle speedprofile, but this can be changed for each scenario by adding:

    Given the speedprofile "car"

you can also modify the speedprofile by using:

    And the speedprofile settings
    | obeyOneways | no |
    | primary     | 50 |

note that 'Given' and 'And' is interchangeable in cucumber, you should just use what creates a natural language flow.

the second flavour of tests simply lists ways and their tags:

Scenario: Simplest possible oneway
    Given the defaults
    Then routability should be
     | highway | oneway | forw | backw |
     | primary |        | x    | x     |
     | primary | yes    | x    |       |
     | primary | -1     |      | x     |

an osm file is constructed, containing an isolated way for each row. columns are converted to tags. forw and backw has special meaning; if an forw column is present the way is tested for forward routability. if an backw row is present, it's tested for backwards routability.

nodes are spaced on grid with origin at 1,1 and a grid size of 100m

Running tests:
You can run all tests, or specific sets or scenarios.

to run all test: cucumber (or 'rake test')
to run test with a specific tag, use: cucumber --tags @restrictions
to run a specific scenario: cucumber features/restrictions.feature:6

routes are valided by parsing the instructions hash. osrm currently often return empty routes, which is one of the main reasons a lot of test are currently failing.

during test, several log files are used. output from osrm-extract and osrm-prepare are directed to test/preprocess.log. output from osrm-routed is directed to test/osrm-routed.log.

tests results are shown in the terminal. green rows indicates a passed route. yellow is an expected, but missing, result, while grey is an unexpected result.

there's an issue with the order failing rows are displayed, which can be somewhat confusing - the failing row is not directly beneauth the expected row.

additional info is logged to fail.log, containing details about the osm data used, the speedprofile, expected and returned routes, as well as actual query and response involved.

there's currently a lot of failing tests. i'm sure some are caused by incorrect test cases or bugs in the cucumber setup. we just need to iron things out.

Rake task:
a few useful rake task are included. you can run 'rake -T' to get a list of rake commands.

for example, you can run 'rake download ' to download and crop osm data into your sandbox folder. curently only a few fixed areas work, including:

kbh: openhagen
dk: denmark

after downloading, you can run 'rake process' to process the downloaded data.

to run your server in the terminal, use 'rake run'. you must download and process data beforehand.

you can also run the server in the background with 'rake up' and 'rake down'. output is directed to osrm-routed.log. again you must download and process data beforehand. note however, that the include index.html file doesn't point to localhost, which is what you need to use the server you run locally.

@DennisOSRM DennisOSRM commented on the diff
DataStructures/ExtractorCallBacks.h
@@ -214,10 +220,6 @@ class ExtractorCallbacks{
return true;
}
- if ( w.direction == _Way::opposite ){
@DennisOSRM Owner

I suspect this line to break oneway=-1

@emiltin Collaborator
emiltin added a note

you're welcome to ignore/revert this change

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@DennisOSRM
Owner

Thanks a lot. One question though, could you move the ini files back to the root directory?

@emiltin
Collaborator

sure no problem. i added the sandbox folder as a place for running osrm, downloading data, etc, but i'm happy to move it all back into the root folder

@emiltin
Collaborator

moved all ini files back to the root folder

@DennisOSRM
Owner

This needs further documentation on how to get it run.

@emiltin
Collaborator

ok. i'll try to write up something. anything particular that you miss?

@DennisOSRM
Owner

A wiki page on how to set up any depencies would be great. Some recipe to get the tests running.

@emiltin
Collaborator

i could also update the brew formula to easily install everything on mac

@DennisOSRM
Owner

sure, I'll then add the steps to do so on Ubuntu.

@emiltin
Collaborator

good idea with the wiki page

@DennisOSRM
Owner

I put up a wiki page with the basic information regarding the tests: https://github.com/DennisOSRM/Project-OSRM/wiki/Cucumber-Test-Suite

@emiltin
Collaborator

thanks a lot

@DennisOSRM
Owner

I am getting the stuff to run on Linux right now. Is it possible to have cucumber create the test directory if it is not yet existing?

@emiltin
Collaborator

yes, but the test folder should be in the repo, since the .stxxl file needs to be there

@emiltin
Collaborator

the reason is that tests need an .stxxl config files with a minimal disk of 1MB, otherwise it takes too long to launch and terminate the binaries

@DennisOSRM
Owner

One odd thing is that cucumber does not properly shutdown the routed process before spawning a new one (at least on Ubuntu).

@emiltin
Collaborator

the file feature/support/hooks.rb specified things to do before and after running each scenario

@emiltin
Collaborator

the code is in https://github.com/emiltin/Project-OSRM/blob/cucumber_cleanup/features/support/launch.rb#L53

it uses the 'ps' system command, and searches for processes with the name 'osrm-routed', then sends it a signal 9 (KILL)

@DennisOSRM
Owner

I don't know the bevavior on Mac, but I guess that

ps -o pid -o state -o ucomm

lists all running processes of a user, right? On Linux it doesn't. So, I get the following output:

$ ps -o pid -o state -o ucomm
  PID S COMMAND
 4715 S bash
17996 R ps

But checking with '-e' gives me:

$ ps -e -o pid -o state -o ucomm | grep osrm
17429 S osrm-routed
@emiltin
Collaborator

from the manual:

 -A      Display information about other users' processes, including those without controlling terminals.
 -e      Identical to -A.

guess we can just add that option?

@DennisOSRM
Owner

There must be something else. Adding -e or -A didn't help too much. In the end it also leaves a dangling osrm-routed process lying around.

@DennisOSRM
Owner

I am not really Ruby-literate, but this short script should output if it detects a running instance of osrm-routed, right?

#!/usr/bin/ruby
def each_process name, &block
  process_list = `ps -A -o pid -o state -o ucomm`
  process_list.scan /(\d+)\s+([^\s]*)\s+(#{name})/ do |pid,state|
  yield pid.to_i, state.strip if ['I','R','S','T'].include? state[0]
end
end

def find_pid name
  each_process(name) { |pid,state| return pid.to_i }
  return -1
end

print "started\n"
print "detected pid: " + find_pid("bash").to_s + "\n"
print "finished.\n"
@DennisOSRM
Owner

So, I played around a bit. On Linux the output of 'ps' is different to the output of Mac OS X. Could you give me a sample output of ps -A -o pid -o state -o ucomm on the Mac's Terminal?

@DennisOSRM
Owner

I am working on automated testing. A cronjob will be executing these tests. Results are then pushed onto the web. The URL is going to be http://project-osrm.org/cucumber.html

Right now, most tests are failing because of the 'ps' issue, but this can be fixed.

@emiltin
Collaborator
~/Project-OSRM$ ps -A -o pid -o state -o ucomm
  PID STAT UCOMM
    1 Ss   launchd         
   10 Ss   kextd           
   11 Ss   UserEventAgent  
   12 Ss   notifyd         
   13 Ss   diskarbitrationd
   15 Ss   ntpd            
   17 SNs  warmd           
   18 Ss   usbmuxd         
   20 Us   special_file_han
   21 Ss   syslogd         
   23 Ss   stackshot       
   24 Ss   securityd       
   26 Ss   revisiond       
   27 Ss   powerd          
   30 Ss   opendirectoryd  
   32 Ss   mds             
   33 Ss   mDNSResponder   
   35 Ss   loginwindow     
   37 Ss   KernelEventAgent
   39 Ss   hidd            
   40 Ss   fseventsd       
   42 Ss   dynamic_pager   
   46 Ss   configd         
   49 Ss   autofsd         
   52 Ss   distnoted       
   56 Ss   coreservicesd   
   85 SNs  netbiosd        
   91 Ss   WindowServer    
   95 Ss   CVMServer       
  105 Ss   coreaudiod      
  108 Ss   logind          
  118 Ss   launchd         
  121 S    UserEventAgent  
  123 S    distnoted       
  134 S    talagent        
  135 S    SystemUIServer  
  136 S    Finder          
  145 S    fontd           
  152 S    pboard          
  301 S    AppleSpell      
  304 SN   warmd_agent     
  305 S    ubd             
  311 S    imagent         
  316 S    AirPort Base Sta
  319 S    postgres        
  324 Ss   postgres        
  325 Ss   postgres        
  326 Ss   postgres        
  327 Ss   postgres        
  664 Ss   filecoordination
  911 S    lsboxd          
 2510 S    AAM Updates Noti
 3177 U    Dock            
 3178 Ss   com.apple.dock.e
 4104 S    cookied         
 4498 Ss   blued           
15508 S    DashboardClient 
45824 S    ssh-agent       
48055 S    PenTabletDriver 
48059 S    ConsumerTouchDri
48062 S    TabletDriver    
60030 S    TextMate        
69750 S    Activity Monitor
69753 Ss   activitymonitord
70526 U    Console         
73289 S    firefox         
74581 SN   mdworker        
74969 S    Terminal        
76095 S    iTunesHelper    
76993 S    plugin-container
81713 S    imklaunchagent  
82032 Ss   launchd         
82035 Ss   xpchelper       
82038 SN   mdworker        
82039 S    distnoted       
82040 Ss   xpchelper       
74971 Ss   login           
74972 S    bash            
81528 T    man             
81529 T    sh              
81530 T    sh              
81534 T    sh              
81536 T    less            
81765 S+   ruby            
81767 S+   osrm-routed     
78905 Ss   login           
78906 S+   bash            
78919 T    ping            
82052 Ss   login           
82053 S    bash            
82067 R+   ps              
@emiltin
Collaborator

maybe you can post what the output is on linux?
an alternative is to use ruby''s build in Process class to fetch the list

@DennisOSRM
Owner

On Linux it is

```~/Project-OSRM$ ps -A -o pid -o state -o ucomm
PID S COMMAND
1 S init
2 S kthreadd
3 S ksoftirqd/0
6 S migration/0
7 S migration/1
9 S ksoftirqd/1
11 S cpuset
12 S khelper
13 S netns
14 S kworker/u:1
15 S sync_supers
16 S bdi-default
17 S kintegrityd
18 S kblockd
19 S ata_sff
20 S khubd
21 S md
23 S khungtaskd
24 S kswapd0
25 S ksmd
26 S khugepaged
27 S fsnotify_mark
28 S ecryptfs-kthrea
29 S crypto
37 S kthrotld
38 S scsi_eh_0
39 S scsi_eh_1
40 S kworker/u:2
41 S scsi_eh_2
42 S scsi_eh_3
226 S scsi_eh_4
228 S scsi_eh_5
229 S scsi_eh_6
230 S scsi_eh_7
278 S flush-8:0
279 S kjournald
328 S upstart-udev-br
330 S udevd
353 S kworker/1:2
544 S krdsd
615 S hd-audio0
753 S upstart-socket-
756 S rpcbind
776 S rpciod
777 S nfsiod
863 S sshd
935 S rpc.gssd
939 S rpc.statd
940 S rsyslogd
953 S rpc.idmapd
962 S automount
1133 S dbus-daemon
1154 S modem-manager
1163 S cupsd
1184 S avahi-daemon
1185 S avahi-daemon
1192 S colord
1199 S getty
1204 S getty
1214 S NetworkManager
1216 S lightdm
1221 S getty
1222 S getty
1224 S getty
1230 S acpid
1245 S atd
1246 S cron
1295 S polkitd
1303 S Xorg
1333 S ntpd
1390 S accounts-daemon
1431 S console-kit-dae
1615 S upowerd
1741 S vmware-usbarbit
1801 S kworker/0:2
1821 S rtkit-daemon
1825 S python
1846 S sh
1848 S osrm-routed
1851 S iked.real
2008 S nscd
2086 S iprt
2089 S udevd
2146 S winbindd
2150 S winbindd
2220 S sendmail-mta
2264 S dhcdbd
2277 S bluetoothd
2281 S l2cap
2284 S krfcommd
2293 S gpsd
2350 S apache2
2392 S getty
2395 S apache2
2396 S apache2
2397 S apache2
2398 S apache2
2399 S apache2
2400 S udevd
2455 S gnome-keyring-d
2464 S gnome-session
2510 S ssh-agent
2513 S dbus-launch
2514 S dbus-daemon
2516 S gvfsd
2521 S gvfs-fuse-daemo
2530 S gconfd-2
2534 S gnome-settings-
2547 S gnome-screensav
2548 S compiz
2554 S pulseaudio
2557 S gconf-helper
2561 S polkit-gnome-au
2563 S blueman-applet
2565 S nautilus
2567 S gnome-fallback-
2568 S nm-applet
2573 S gvfs-gdu-volume
2577 S skype
2579 S bluetooth-apple
2581 S udisks-daemon
2582 S udisks-daemon
2585 S gvfs-gphoto2-vo
2596 S xfconfd
2600 S gvfs-afc-volume
2620 S gdu-notificatio
2622 S obex-data-serve
2624 S gvfsd-trash
2633 S gvfsd-burn
2645 S sh
2646 S unity-window-de
2648 S bamfdaemon
2660 S gvfsd-metadata
2693 S unity-panel-ser
2724 S applet.py
2726 S indicator-sessi
2735 S indicator-sound
2736 S indicator-messa
2739 S indicator-appli
2771 S ubuntuone-syncd
2799 S gsd-printer
2804 S dconf-service
2810 S thunderbird-bin
2822 S firefox
2826 S unity-applicati
2828 S unity-files-dae
2830 S unity-music-dae
2856 S unity-musicstor
2895 S zeitgeist-daemo
2898 S cat
2900 S zeitgeist-datah
2912 S update-notifier
2998 S xchat
3035 S plugin-containe
3054 S npviewer.bin
3080 S /usr/bin/termin
3092 Z /usr/bin/termin
3093 S bash
4715 S bash
4778 R ps
19419 S gvfsd-http
19552 S pidgin
27298 S plugin-containe
27317 S npviewer.bin
27336 Z acroread
27809 S kworker/0:0
30276 S kworker/1:1
30344 S eclipse
30358 S java
32566 S kworker/1:0
32662 S geany
32665 Z geany
32666 S bash
32683 S kworker/0:1
32749 S xfce4-notifyd


Using the builtin Process class is perhaps the most portable way.
@emiltin
Collaborator

i'm noticing that states can include "+", so it might be the fltering code that's wrong:
['I','R','S','T'].include? state[0]

@emiltin
Collaborator

yeah Process might be the way forward

@emiltin
Collaborator

another alternative: killall
http://en.wikipedia.org/wiki/Killall

@emiltin
Collaborator

changed the code to use a ruby gem to fetch process list.

note: you need to rerun 'bundle install' in the root folder, to get the sys-proctable gem installed.

@DennisOSRM
Owner

This gives better results, but still not perfect.

32 scenarios (15 failed, 17 passed)
88 steps (15 failed, 73 passed)
1m25.016s

How many tests are supposed to fail right now?

@emiltin
Collaborator

lets start simple. when i run "cucumber --tags @basic" i get:

7 scenarios (3 failed, 4 passed)
21 steps (3 failed, 18 passed)

as far as i can tell, these are legitimate failures, caused by odd osrm behaviour?

@emiltin
Collaborator

you can also try "cucumber --tags @smallest". this runs a single, very simple test. this should pass without problems.

@DennisOSRM
Owner

7 scenarios (2 failed, 5 passed)
21 steps (2 failed, 19 passed)

@DennisOSRM
Owner

cucumber --tags @smallest" runs fine.

@emiltin
Collaborator

or run a single test with "cucumber features/basic.feature:54". the first row of this one currently fails for me, returning bc, instead of ab,bc

(Scenario: Two ways connected in a straight line)

@emiltin
Collaborator

i moved one scenario out of the @basic scope, so i think we're now getting consistent results?

@emiltin
Collaborator

note that test/fail.log will contain additional info about failing scenarios.

@DennisOSRM
Owner

I did not yet look at all the failing tests, but should they all run through?

@emiltin
Collaborator

no they don't all run through... some could be due to incorrect tests of course. as i said, we just need to iron things out. we can perhaps look at one failing test and see what's going on.

@emiltin
Collaborator

for example: Two ways connected in a straight line.
going from a to c returns bc. should this be ab,bc as indicated in the test?

@emiltin
Collaborator

perhaps we should use ab-bc instead of ab,bc. might avoid a few misunderstandings...

@DennisOSRM
Owner

I am close to preparing the final push to master repo. We should start changing, discussing failing tests afterwards.

@emiltin
Collaborator

makes sense

@emiltin
Collaborator

you might want to check on that thing with oneway=-1. as i said, you're welcome to revert/ignore that change if you want

@DennisOSRM
Owner

I ignored it ;-)

@emiltin
Collaborator

good choice :-)

@emiltin
Collaborator

i'm about to call it a day, but will be back tomorrow morning

@DennisOSRM
Owner

I'll keep on hacking on this for the coming two hours. The ambition to get it work kicked in.

@DennisOSRM
Owner
$ cucumber --tags @basic
7 scenarios (7 passed)
21 steps (21 passed)

There's progress!

@emiltin
Collaborator

nice!!

later

@DennisOSRM DennisOSRM closed this
@Zhdanovich Zhdanovich referenced this pull request from a commit in Zhdanovich/Project-OSRM
DennisOSRM Adds the cucumber test framework referenced in issues #26, #95, #114 f91f17a
@Zhdanovich Zhdanovich referenced this pull request from a commit in Zhdanovich/Project-OSRM
@emiltin emiltin Adds the cucumber test framework referenced in issues #26, #95, #114 ced10e2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 11, 2012
  1. @emiltin

    cleaned up cucumber tests

    emiltin authored
Commits on Feb 13, 2012
  1. @emiltin
  2. @emiltin
  3. @emiltin

    updated tests for oneway

    emiltin authored
Commits on Feb 14, 2012
  1. @emiltin

    use ruby to fetch process list

    emiltin authored
Something went wrong with that request. Please try again.