Flexible and scalable website testing tool.
-
David Brady – github@shinybit.com
-
Tim Harper – tim.harper@leadmediapartners.com
-
James Britt – james@neurogami.com
-
JT Zemp – jtzemp@gmail.com
-
Joe Tanner – joe.tanner@leadmediapartners.com
-
seph – seph@directionless.org
TourBus is an intelligent website load testing tool. Allows for complicated testing scenarios including filling out forms, following redirects, handling cookies, and following links–all of the things you’d normally associate with a regression suite or integration testing tool. The difference is that TourBus also scales concurrently, and you can perform hundreds of complicated regression tests simultaneously in order to thoroughly load test your website.
It uses Webrat::Mechanize to run the browsing session, so you get the load testing you want, and all the sweetness of Webrat to write your tests in.
In December of 2010, it was reworked to change the data model. Now it uses simpler thread&dispatch model, and has a tweaked data model. This allows it to represent other workflows.
TourBus was written for flexibility and scalability in a website testing tool. The existing tools tend to provide one, but not the other. Selenium is ultraflexible but limited to the number of browsers you can have open at once, while Apache Bench is powerful and fast but limited to simple tests.
TourBus lets you define complicated paths through your website, then execute those paths concurrently for stress testing.
It’s worth taking a few minutes to understand the TourBus abstraction model. It’ll help understand the various things.
The main piece is a ‘Tourist`, this represents a common web visitor’s workflow.
Moving downwards, a ‘Tourist` goes on a series of `Tours`, and timing data is collected per tour. A `Tour` may have multiple things inside, but we don’t talk about those.
Moving upwards, there are ‘Guides`. These control how many `Tourists` are operating concurrently. A Guide probabilistically picks a Tourist, takes them on their chosen Tours, reports timing data back to the TourBus, and then repeats.
TourBus (stats are stored here)
-> Guide (one per concurrent thread) -> Tourist (has a scheduled list of tours) -> tours
We want to simulate N concurrent users, each operating independantly.
What good is testing without results? As this is designed to be run thousands of time, there’s a lot of data. Right now, we’re just storing data and leaving the presentation to other tools.
Since everyone loves nosql, and that allows us to skip on schema definitions, let’s go with that. There are several different messages.
(runid is the time the run started. It’s unique enough for us)
The main logs:
runid, concurrancy, tourist_type,
tour_num, tour_name, start_time, end_time, elapsed_time, status, messages
runid, concurrancy, tourist_type,
"total", start_time, end_time, elapsed_time, status
We might add an additional thread to note machine load. Since everyone infrastructure is different, this is mostly left as an exercise to the reader. But the log format might look like:
runid, concurrancy, “monitoring”, “db”, “load”, time, loadavg
To see TourBus in action, you need to write scripts. For lack of a better name, these are called Tours.
-
Make a folder called tourists and put a file in it called simple.rb. In it write:
class Simple < Tourist WEIGHT = 10 def tour_homepage visit "http://#{@host}/" assert_contain "My Home Page" end end
-
Files in ./tourists should have classes that match their names. E.g. “class BigHairyTest < Tourist” belongs in ./tourists/big_hairy_test.rb
-
Think Test::Unit. tour_* methods will be found automagically. setup() and teardown() methods will be executed at the appropriate times.
You want to invoke tourbus
from the parent directory of the @tourists/@ folder.
For example, if you have this project tree …
`-- contact_app |-- README.rdoc |-- contact_app.rb `-- tourists |-- simple.rb `-- tourbus.yml
… then you execute tourbus
from the contact_app/
directory.
tourbus -c 2 -n 3 simple
That will run the simple.rb
tourist file.
It will create 2 concurrent Tour runners, each of which will run all of the methods in Simple three times.
-
You can specify multiple tourists.
tourbus -c 2 -n 3 simple1 simple2 simple3
-
If you don’t specify a tour, all tours in ./tourists will be run.
-
tourbus –help will give you more information.
-
You can run tours and filter given tests.
tourbus -c 2 -n 3 simple -t test_login,test_logout
Note that if you specify multiple tours and filter tests, the filtered tests will be run on all tours specified. If you do not specify a tour, the filtered tests will be run on all tours found in the ./tourists
folder.
On the webserver, you can type
tourwatch -c 4
To begin running tourwatch. It’s basically a stripped-down version of top with cheesy text graphs. (TourWatch’s development cycles were included in the 2 days for TourBus.)
-
The -c option is for the total number of cores on the server. The top app will cheerfully report a process as taking 392% CPU if it is using 98% of four cores. This option is only necessary for making the little text graphs scale correctly.
-
You can choose which processes to watch by passing a csv to -p:
tourwatch -p ruby,mongrel
Each process name is a partial regexp, so the above would match mongrel AND mongrel_rails, etc.
-
tourwatch –help will give you more information.
-
save_and_open_page
TourBus began life as a 2-day throwaway app. It is definitely an app whose development provides many opportunities for open-source contributors to make improvements. It is chock-full of brutal hacks, duplications, oversights, and kludges.
-
If you give a tour a name that is pluralized, it won’t work. This is probably a bug worth fixing. The reason for it is that we take file names and “classify” them, and e.g. “ranking_reports” becomes “RankingReport”, not “RankingReports”. This is an artifact of borrowing from Rails’ activesupport libs and should probably be fixed.
-
Mechanize 0.8 doesn’t always play well together with TourBus. If you get “connection refused” socket errors, try upgrading to Mechanize 0.9.
-
JRuby doesn’t play well with Nokogiri. I have set the html_parser to use hpricot, which should work around the issue for now.
-
There are no specs. Yikes! This is to my eternal shame because I’m sort of a testing freak. Because TourBus WAS a testing tool, I didn’t put tests on it. I haven’t put tests on it yet because I’m not sure how to go about it. Instead of exercising a web app with a test browser, we need to exercise a test browser with… um… a web app? (dbrady notes: Now that we have a contact_app, we could try writing some specs and features to run tourbus against it.)
-
Web-Sickle is another internal app, written by Tim Harper, that works “well enough”. Until I open-sourced this project, it was a submodule in the app. We wanted to keep TourBus extensions separate from WebSickle itself, so there’s a lot of code in Runner that really belongs in WebSickle.
-
Documentation is <strike>horrible</strike> merely quite bad.
-
There’s not much in the way of examples, either. When I removed all the LMP-specific code, all of the examples went with it. Sorry about that. Coming soon.
-
I’d like to beef up the example contact app to show more of TourBus than the simplest possible path. Adding in another page or two, and then adding an additional tour or two would make it more apparent to new users that you can do things like run multiple tours at once. Also, having more than one test_ method in simple.rb would let us demonstrate test filtering as well. (Be aware that at present [2009-04-17], webrat still does not play well with Sinatra sessions so there would be complications. dbrady’s fork of webrat combines jferris’ fix with the latest webrat, and will be maintained until main webrat includes the feature. That fork is at github.com/dbrady/webrat)
-
I’d like to remove WebSickle and replace it with Webrat. There is a webrat branch on the main fork (github.com/dbrady/tourbus) that is 90% complete. Once that’s done we can start massaging the API to be a little more friendly. [done (but now that it is, it needs a refactoring–Tour should probably inherit from Webrat::Mechanize, not delegate to it.)]
-
Tim Harper camped at my place for a day fixing bugs in WebSickle as I exercised more and more new bits of it. Thanks, dude.
-
Lead Media Partners paid me to write TourBus, then let me open source it. How much do they rock? All the way to 11, that’s how much they rock.
-
James Britt jumped on this and revived it as it was gathering dust. Thanks!
-
JT Zemp added before_tour, after_tour. Thanks!
MIT. See the license file.