Skip to content
Browse files

Initial commit

  • Loading branch information...
0 parents commit 8b2161959ca99e28ee5d2cf170c346df6b5ee6fa @dbrady committed Jan 1, 2009
1 .gitignore
@@ -0,0 +1 @@
+doc/*
22 MIT-LICENSE
@@ -0,0 +1,22 @@
+Copyright (c) 2008-2009 David Brady github@shinybit.com
+
+Permission is hereby granted, free of charge, to any person
+obtaining a copy of this software and associated documentation
+files (the "Software"), to deal in the Software without
+restriction, including without limitation the rights to use,
+copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the
+Software is furnished to do so, subject to the following
+conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
+OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
+HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+OTHER DEALINGS IN THE SOFTWARE.
139 README.txt
@@ -0,0 +1,139 @@
+= TourBus
+
+Flexible and scalable website testing tool.
+
+== Authors
+
+* David Brady -- david.brady@leadmediapartners.com
+* Tim Harper -- tim.harper@leadmediapartners.com
+
+== General Info
+
+TourBus is an intelligent website load testing tool. Allows for
+complicated testing scenarios including filling out forms, following
+redirects, handling cookies, and following links--all of the things
+you'd normally associate with a regression suite or integration
+testing tool. The difference is that TourBus also scales concurrently,
+and you can perform hundreds of complicated regression tests
+simultaneously in order to thoroughly load test your website.
+
+== Motivation
+
+I started writing TourBus because I needed flexibility and scalability
+in a website testing tool, and the extant tools all provided one but
+not the other. Selenium is ultraflexible but limited to the number of
+browsers you can have open at once, while Apache Bench is powerful and
+fast but limited to simple tests.
+
+TourBus lets you define complicated paths through your website, then
+execute those paths concurrently for stress testing.
+
+== Example
+
+To see TourBus in action, you need to write scripts. For lack of a
+better name, these are called Tours.
+
+=== Example Tour
+
+* Make a folder called tours and put a file in it called simple.rb. In
+ it write:
+
+ class Simple < Tour
+ def test_homepage
+ open_page "http://#{@host}/"
+ assert_page_body_contains "My Home Page"
+ end
+ end
+
+* Files in ./tours should have classes that match their names. E.g.
+ "class BigHairyTest < Tour" belongs in ./tours/big_hairy_test.rb
+
+* Think Test::Unit. test_* methods will be found automagically.
+ setup() and teardown() methods will be executed at the appropriate
+ times.
+
+=== Example TourBus Run
+
+tourbus -c 2 -n 3 simple
+
+This will create 2 concurrent Tour runners, each of which will run all
+of the methods in Simple three times.
+
+* You can specify multiple tours.
+
+* If you don't specify a tour, all tours in ./tours will be run.
+
+* tourbus --help will give you more information.
+
+=== Example TourWatch Run
+
+On the webserver, you can type
+
+tourwatch -c 4
+
+To begin running tourwatch. It's basically a stripped-down version of
+top with cheesy text graphs. (TourWatch's development cycles were
+included in the 2 days for TourBus.)
+
+* The -c option is for the total number of cores on the server. The
+ top app will cheerfully report a process as taking 392% CPU if it is
+ using 98% of four cores. This option is only necessary for making
+ the little text graphs scale correctly.
+
+* You can choose which processes to watch by passing a csv to -p:
+
+ tourwatch -p ruby,mongrel
+
+ Each process name is a partial regexp, so the above would match
+ mongrel AND mongrel_rails, etc.
+
+* tourwatch --help will give you more information.
+
+== History and Status
+
+TourBus began life as a 2-day throwaway app. It is definitely an app
+whose development provides many opportunities for open-source
+contributors to make improvements. It is chock-full of brutal hacks,
+duplications, oversights, and kludges.
+
+== Hacks, Kludges, Known Issues, and Piles of Steaming Poo
+
+* Mechanize 0.8 doesn't always play well together with TourBus. If you
+ get "connection refused" socket errors, try upgrading to Mechanize
+ 0.9.
+
+* JRuby doesn't play well with Nokogiri. I have set the html_parser to
+ use hpricot, which should work around the issue for now.
+
+* There are no specs. Yikes! This is to my eternal shame because I'm
+ sort of a testing freak. Because TourBus *WAS* a testing tool, I
+ didn't put tests on it. I haven't put tests on it yet because I'm
+ not sure how to go about it. Instead of exercising a web app with a
+ test browser, we need to exercise a test browser with... um... a web
+ app?
+
+* Web-Sickle is another internal app, written by Tim Harper, that
+ works "well enough". Until I open-sourced this project, it was a
+ submodule in the app. We wanted to keep TourBus extensions separate
+ from WebSickle itself, so there's a lot of code in Runner that
+ really belongs in WebSickle.
+
+* Documentation is horrible.
+
+* There's not much in the way of examples, either. When I removed all
+ the LMP-specific code, all of the examples went with it. Sorry about
+ that. Coming soon.
+
+== Credits
+
+* Tim Harper camped at my place for a day fixing bugs in WebSickle as
+ I exercised more and more new bits of it. Thanks, dude.
+
+* Lead Media Partners paid me to write TourBus, then let me open
+ source it. How much do they rock? All the way to 11, that's how much
+ they rock.
+
+== License
+
+MIT. See the license file.
+
13 Rakefile
@@ -0,0 +1,13 @@
+# Add your own tasks in files placed in lib/tasks ending in .rake, for
+# example lib/tasks/capistrano.rake, and they will automatically be
+# available to Rake.
+
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+
+Rake::RDocTask.new do |rd|
+ rd.main = "README.txt"
+ rd.rdoc_dir = "doc"
+ rd.rdoc_files.include("README.txt", "bin/*", "lib/**/*.rb")
+end
29 bin/tourbus
@@ -0,0 +1,29 @@
+#!/usr/bin/env ruby
+
+require File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib', 'common'))
+require 'trollop'
+require_all_files_in_folder 'tours'
+
+opts = Trollop.options do
+ opt :host, "Remote hostname to test", :default => "localhost:3000"
+ opt :concurrency, "Number of simultaneous runs to perform", :type => :integer, :default => 1
+ opt :number, "Number of times to run the tour (in each concurrent step, so -c 10 -n 10 will run the tour 100 times)", :type => :integer, :default => 1
+ opt :verbose, "Run in verbose mode", :type => :boolean, :default => false
+ opt :list, "List tours and runs available. If tours or runs are included, filters the list", :type => :boolean, :default => nil
+end
+
+tours = if ARGV.empty?
+ Dir[File.join('.', 'tours', '*.rb')].map {|f| File.basename(f, ".rb")}
+ else
+ ARGV
+ end
+
+if opts[:list]
+ Tour.tours(ARGV).each do |tour|
+ puts tour
+ puts Tour.tests(tour).map {|test| " #{test}"}
+ end
+else
+ puts Benchmark.measure { TourBus.new(opts[:host], opts[:concurrency], opts[:number], tours).run }
+end
+
52 bin/tourwatch
@@ -0,0 +1,52 @@
+#!/usr/bin/env ruby
+
+# tourwatch - cheap monitor program for tourbus
+#
+# Notes:
+#
+# tourwatch is a cheap logger program for tourbus. It runs on the
+# targeted server and monitors cpu and memory usage of webserver
+# processes. It's a moderately quick hack: I have a 2-hour budget to
+# write and debug the whole thing and here I am wasting time by
+# starting with documentation. This is because I figure the chance of
+# this program needing maintenance in the next 6 months to be well
+# over 100%, and the poor guy behind me (Hey, that's you! Hi.) will
+# need to know why tourwatch is so barebones.
+#
+# So. TourWatch runs on the target server, collects top information
+# every second, and logs it to file. End of story. "Automation" is
+# handled by the meat cloud (Hey, that's you! Hi.) when the maintainer
+# starts and stops the process manually. Report collection is handled
+# by you reading the logfiles in a terminal. Report aggregation is
+# handled by you aggregating the reports. Yes, there's a theme here.
+#
+# TODO:
+#
+# - Remote reporting? Send log events to main log server?
+#
+# - If we logged to a lightweight database like sqlite3, we could do
+# some clever things like track individual pids and process groups.
+# This would let us track, e.g., aggregate apache stress as well as
+# rogue mongrels. I'm not doing this now because it will require
+# writing something to read and parse the previous information. For
+# now, we'll leave it up to the user (Hey, that's you! Hi.) to parse
+# the logfiles.
+#
+# - Tweak output format. Currently it's crap. I don't think we need
+# dynamic templating or anything, but it might be nice to improve
+# the existing formats.
+
+require File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib', 'common'))
+require 'trollop'
+require 'tour_watch'
+
+opts = Trollop.options do
+ opt :outfile, "Logfile name (default to STDOUT)", :type => :string, :default => nil
+ opt :processes, "csv of processes to monitor", :type => :string, :default => nil
+ opt :cores, "number of cores present (max CPU% is number of cores * 100)", :type => :integer, :default => 4
+ opt :mac, "Set if running on MacOSX. The Mac top command is different than linux top.", :type => :boolean, :default => false
+end
+
+TourWatch.new(opts).run
+
+
31 lib/common.rb
@@ -0,0 +1,31 @@
+# common.rb - Common settings, requires and helpers
+unless defined? TOURBUS_LIB_PATH
+ TOURBUS_LIB_PATH = File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib'))
+ $:<< TOURBUS_LIB_PATH unless $:.include? TOURBUS_LIB_PATH
+end
+
+require 'rubygems'
+
+gem 'mechanize', ">= 0.8.5"
+gem 'trollop', ">= 1.10.0"
+gem 'faker', '>= 0.3.1'
+
+# TODO: I'd like to remove dependency on Rails. Need to see what all
+# we're using (like classify) and remove each dependency individually.
+require 'activesupport'
+
+require 'monitor'
+require 'faker'
+require 'web-sickle/init'
+require 'tour_bus'
+require 'runner'
+require 'tour'
+
+
+class TourBusException < Exception; end
+
+def require_all_files_in_folder(folder, extension = "*.rb")
+ for file in Dir[File.join('.', folder, "**/#{extension}")]
+ require file
+ end
+end
67 lib/runner.rb
@@ -0,0 +1,67 @@
+require 'monitor'
+require 'common'
+
+class Runner
+ attr_reader :host, :tours, :number, :runner_type, :runner_id
+
+ def initialize(host, tours, number, runner_id)
+ @host, @tours, @number, @runner_id = host, tours, number, runner_id
+ @runner_type = self.send(:class).to_s
+ log("Ready to run #{@runner_type}")
+ end
+
+ # Dispatches to subclass run method
+ def run_tours
+ runs,passes,fails,errors = 0,0,0,0
+ 1.upto(number) do |num|
+ log("Starting #{@runner_type} run #{num}/#{number}")
+ begin
+ @tours.each do |tour|
+ runs += 1
+ tour = Tour.make_tour(tour,@host,@tours,@number,@runner_id)
+ tour.tests.each do |test|
+ tour.run_test test
+ end
+ end
+ passes += 1
+ rescue TourBusException => e
+ log("***********************************")
+ log("********** ERROR IN RUN! **********")
+ log("***********************************")
+ log e.message
+ e.backtrace.each do |trace|
+ log trace
+ end
+ fails += 1
+ rescue WebsickleException => e
+ log("***********************************")
+ log("********** ERROR IN RUN! **********")
+ log("***********************************")
+ log e.message
+ e.backtrace.each do |trace|
+ log trace
+ end
+ fails += 1
+ rescue Exception => e
+ log("***********************************")
+ log("********** ERROR IN RUN! **********")
+ log("***********************************")
+ log e.message
+ e.backtrace.each do |trace|
+ log trace
+ end
+ errors += 1
+ end
+ log("Finished #{@runner_type} run #{num}/#{number}")
+ end
+ log("Finished all #{@runner_type} runs.")
+ [runs,passes,fails,errors]
+ end
+
+ protected
+
+ def log(message)
+ puts "#{Time.now.strftime('%F %H:%M:%S')} Runner ##{@runner_id}: #{message}"
+ end
+end
+
110 lib/tour.rb
@@ -0,0 +1,110 @@
+require 'monitor'
+require 'common'
+
+# A tour is essentially a test suite file. A Tour subclass
+# encapsulates a set of tests that can be done, and may contain helper
+# and support methods for a given task. If you have a two or three
+# paths through a specific area of your website, define a tour for
+# that area and create test_ methods for each type of test to be done.
+
+class Tour
+ include WebSickle
+ attr_reader :host, :tours, :number, :tour_type, :tour_id
+
+ def initialize(host, tours, number, tour_id)
+ @host, @tours, @number, @tour_id = host, tours, number, tour_id
+ @tour_type = self.send(:class).to_s
+ end
+
+ def setup
+ end
+
+ def teardown
+ end
+
+ # Lists tours in tours folder. If a string is given, filters the
+ # list by that string. If an array of filter strings is given,
+ # returns items that match ANY filter string in the array.
+ def self.tours(filter=[])
+ filter = [filter].flatten
+ # All files in tours folder, stripped to basename, that match any item in filter
+ Dir[File.join('.', 'tours', '**', '*.rb')].map {|fn| File.basename(fn, ".rb")}.select {|fn| filter.size.zero? || filter.any?{|f| fn =~ /#{f}/}}
+ end
+
+ def self.tests(tour_name)
+ Tour.make_tour.tests
+ end
+
+ # Factory method, creates the named child class instance
+ def self.make_tour(tour_name,host,tours,number,tour_id)
+ tour_name.classify.constantize.new(host,tours,number,tour_id)
+ end
+
+ # Returns list of tests in this tour. (Meant to be run on a subclass
+ # instance; returns the list of tests available).
+ def tests
+ methods.grep(/^test_/).map {|m| m.sub(/^test_/,'')}
+ end
+
+ def run_test(test_name)
+ test = "test_#{test_name}"
+ raise TourBusException.new("run_test couldn't run test '#{test_name}' because this tour did not respond to :#{test}") unless respond_to? test
+ setup
+ send test
+ teardown
+ end
+
+ protected
+
+ def log(message)
+ puts "#{Time.now.strftime('%F %H:%M:%S')} Tour ##{@tour_id}: #{message}"
+ end
+
+ # given "portal", opens "http://#{@host}/portal"
+ def open_site_page(path)
+ open_page "http://#{@host}/#{path}"
+ end
+
+ def dump_form
+ log "Dumping Forms:"
+ page.forms.each do |form|
+ puts "Form: #{form.name}"
+ puts '-' * 20
+ (form.fields + form.radiobuttons + form.checkboxes + form.file_uploads).each do |field|
+ puts " #{field.name}"
+ end
+ end
+ end
+
+ # True if uri ends with the string given. If a regex is given, it is
+ # matched instead.
+ #
+ # TODO: Refactor me--these were separated out back when Websickle
+ # was a shared submodule and we couldn't pollute it. Now that it's
+ # frozen these probably belong there.
+ def assert_page_uri_matches(uri)
+ case uri
+ when String:
+ raise WebsickleException, "Expected page uri to match String '#{uri}' but did not. It was #{page.uri}" unless page.uri.to_s[-uri.size..-1] == uri
+ when Regexp:
+ raise WebsickleException, "Expected page uri to match Regexp '#{uri}' but did not. It was #{page.uri}" unless page.uri.to_s =~ uri
+ end
+ log "Page URI ok (#{page.uri} matches: #{uri})"
+ end
+
+ # True if page contains (or matches) the given string (or regexp)
+ #
+ # TODO: Refactor me--these were separated out back when Websickle
+ # was a shared submodule and we couldn't pollute it. Now that it's
+ # frozen these probably belong there.
+ def assert_page_body_contains(pattern)
+ case pattern
+ when String:
+ raise WebsickleException, "Expected page body to contain String '#{pattern}' but did not. It was #{page.body}" unless page.body.to_s.index(pattern)
+ when Regexp:
+ raise WebsickleException, "Expected page body to match Regexp '#{pattern}' but did not. It was #{page.body}" unless page.body.to_s =~ pattern
+ end
+ log "Page body ok (matches #{pattern})"
+ end
+end
+
84 lib/tour_bus.rb
@@ -0,0 +1,84 @@
+require 'benchmark'
+
+class TourBus < Monitor
+ attr_reader :host, :concurrency, :number, :tours, :runs, :passes, :fails, :errors, :benchmarks
+
+ def initialize(host="localhost", concurrency=1, number=1, tours=[])
+ @host, @concurrency, @number, @tours = host, concurrency, number, tours
+ @runner_id = 0
+ @runs, @passes, @fails, @errors = 0,0,0,0
+ super()
+ end
+
+ def next_runner_id
+ synchronize do
+ @runner_id += 1
+ end
+ end
+
+ def update_stats(runs,passes,fails,errors)
+ synchronize do
+ @runs += runs
+ @passes += passes
+ @fails += fails
+ @errors += errors
+ end
+ end
+
+ def update_benchmarks(bm)
+ synchronize do
+ @benchmarks = @benchmarks.zip(bm).map { |a,b| a+b}
+ end
+ end
+
+ def runners(filter=[])
+ # All files in tours folder, stripped to basename, that match any item in filter
+ Dir[File.join('.', 'tours', '**', '*.rb')].map {|fn| File.basename(fn, ".rb")}.select {|fn| filter.size.zero? || filter.any?{|f| fn =~ /#{f}/}}
+ end
+
+ def run
+ threads = []
+ started = Time.now.to_f
+ concurrency.times do |conc|
+ log "Starting #{concurrency} runners to run #{tours.size} tours #{number} times (for a total of #{tours.size*concurrency*number} times)"
+ threads << Thread.new do
+ runner_id = next_runner_id
+ runs,passes,fails,errors,start = 0,0,0,0,Time.now.to_f
+ bm = Benchmark.measure do
+ runner = Runner.new(@host, @tours, @number, runner_id)
+ runs,passes,fails,errors = runner.run_tours
+ update_stats runs, passes, fails, errors
+ end
+ log "Runner Finished!"
+ log "Runner finished in %0.3f seconds" % (Time.now.to_f - start)
+ log "Runner Finished! runs,passes,fails,errors: #{runs},#{passes},#{fails},#{errors}"
+ log "Benchmark for runner #{runner_id}: #{bm}"
+ end
+ end
+ log "All Runners started!"
+ threads.each {|t| t.join }
+ finished = Time.now.to_f
+ log '-' * 80
+ log "All Runners finished."
+ log "Total Runs: #{@runs}"
+ log "Total Passes: #{@passes}"
+ log "Total Fails: #{@fails}"
+ log "Total Errors: #{@errors}"
+ log "Elapsed Time: #{finished - started}"
+ log "Speed: %5.3f v/s" % (@runs / (finished-started))
+ log '-' * 80
+ if @fails > 0 || @errors > 0
+ log '********************************************************************************'
+ log '********************************************************************************'
+ log ' !! THERE WERE FAILURES !!'
+ log '********************************************************************************'
+ log '********************************************************************************'
+ end
+ end
+
+ def log(message)
+ puts "#{Time.now.strftime('%F %H:%M:%S')} TourBus: #{message}"
+ end
+
+end
+
88 lib/tour_watch.rb
@@ -0,0 +1,88 @@
+class TourWatch
+ attr_reader :processes
+
+ def initialize(options={})
+ @processes = if options[:processes]
+ options[:processes].split(/,/) * '|'
+ else
+ "ruby|mysql|apache|http|rails|mongrel"
+ end
+ @cores = options[:cores] || 4
+ @logfile = options[:outfile]
+ @mac = options[:mac]
+ end
+
+ def stats
+ top = @mac ? top_mac : top_linux
+ lines = []
+ @longest = Hash.new(0)
+ top.each_line do |line|
+ name,pid,cpu = fields(line.split(/\s+/))
+ lines << [name,pid,cpu]
+ @longest[:name] = name.size if name.size > @longest[:name]
+ @longest[:pid] = pid.to_s.size if pid.to_s.size > @longest[:pid]
+ end
+ lines
+ end
+
+ def fields(parts)
+ @mac ? fields_mac(parts) : fields_linux(parts)
+ end
+
+ # Note: MacOSX is so awesome I just cacked. Top will report 0.0% cpu
+ # the first time you run top, every time. The only way to get actual
+ # CPU% here is to wait for it to send another page and then throw
+ # away the first page. Isn't that just awesome?!? I KNOW!!!
+ def top_mac
+ top = `top -l 1 | grep -E '(#{@processes})'`
+ end
+
+ def fields_mac(fields)
+ name,pid,cpu = fields[1], fields[0].to_i, fields[2].to_f
+ end
+
+ def top_linux
+ top = `top -bn 1 | grep -E '(#{@processes})'`
+ end
+
+
+ def fields_linux(fields)
+ # linux top isn't much smarter. It spits out a blank field ahead
+ # of the pid if the pid is too short, which makes the indexes
+ # shift off by one.
+ a,b,c = if fields.size == 13
+ [-1,1,9]
+ else
+ [-1,0,8]
+ end
+ name,pid,cpu = fields[a], fields[b].to_i, fields[c].to_f
+ end
+
+
+ def run()
+ while(true)
+ now = Time.now.to_i
+ if @time != now
+ log '--'
+ lines = stats
+ lines.sort! {|a,b| a[1]==b[1] ? a[2]<=>b[2] : a[1]<=>b[1] }
+ lines.each do |vars|
+ vars << bargraph(vars[2], 100 * @cores)
+ log "%#{@longest[:name]}s %#{@longest[:pid]}d CPU: %6.2f%% [%-40s]" % vars
+ end
+ end
+ sleep 0.1
+ @time = now
+ end
+ end
+
+ def bargraph(value, max=100, length=40, on='#', off='.')
+ (on * (([[value, 0].max, max].min * length) / max).to_i).ljust(length, off)
+ end
+
+ def log(message)
+ msg = "#{Time.now.strftime('%F %H:%M:%S')} TourWatch: #{message}"
+ puts msg
+ File.open(@logfile, "a") {|f| f.puts msg } if @logfile
+ end
+end
1 lib/web-sickle/.gitignore
@@ -0,0 +1 @@
+.DS_Store
4 lib/web-sickle/README
@@ -0,0 +1,4 @@
+WebSickle
+=========
+
+Description goes here
17 lib/web-sickle/init.rb
@@ -0,0 +1,17 @@
+require 'rubygems'
+gem 'mechanize', ">= 0.7.6"
+gem "hpricot", ">= 0.6"
+$: << File.join(File.dirname(__FILE__), 'lib')
+
+require 'hpricot'
+require 'mechanize'
+
+WWW::Mechanize.html_parser = Hpricot
+
+require 'web_sickle'
+require "assertions"
+require "hash_proxy"
+require "helpers/asp_net"
+require "helpers/table_reader"
+
+Hpricot.buffer_size = 524288
51 lib/web-sickle/lib/assertions.rb
@@ -0,0 +1,51 @@
+class WebSickleAssertionException < Exception; end
+
+module WebSickle::Assertions
+ def assert_equals(expected, actual, message = nil)
+ unless(expected == actual)
+ report_error <<-EOF
+Error: Expected
+#{expected.inspect}, but got
+#{actual.inspect}
+#{message}
+EOF
+ end
+ end
+
+ def assert_select(selector, message)
+ assert_select_in(@page, selector, message)
+ end
+
+ def assert_no_select(selector, message)
+ assert_no_select_in(@page, selector, message)
+ end
+
+ def assert_select_in(content, selector, message)
+ report_error("Error: Expected selector #{selector.inspect} to find a page element, but didn't. #{message}") if (content / selector).blank?
+ end
+
+ def assert_no_select_in(content, selector, message)
+ report_error("Error: Expected selector #{selector.inspect} to not find a page element, but did. #{message}") unless (content / selector).blank?
+ end
+
+ def assert_contains(left, right, message = nil)
+ (right.is_a?(Array) ? right : [right]).each do | item |
+ report_error("Error: Expected #{left.inspect} to contain #{right.inspect}, but didn't. #{message}") unless left.include?(item)
+ end
+ end
+
+ def assert(passes, message = nil)
+ report_error("Error: expected true, got false. #{message}") unless passes
+ end
+
+ def assert_link_text(link, text)
+ case text
+ when String
+ assert_equals(link.text, text)
+ when Regexp
+ assert(link.text.match(text))
+ else
+ raise ArgumentError, "Don't know how to assert an object like #{text.inspect} - expected: Regexp or String"
+ end
+ end
+end
9 lib/web-sickle/lib/hash_proxy.rb
@@ -0,0 +1,9 @@
+class HashProxy
+ def initialize(options = {})
+ @set = options[:set]
+ @get = options[:get]
+ end
+
+ def [](key); @get && @get.call(key); end
+ def []=(key, value); @set && @set.call(key, value); end
+end
16 lib/web-sickle/lib/helpers/asp_net.rb
@@ -0,0 +1,16 @@
+module WebSickle::Helpers
+module AspNet
+ def asp_net_do_postback(options)
+ target_element = case
+ when options[:button]
+ find_button(options[:button])
+ when options[:field]
+ find_field(options[:field])
+ else
+ nil
+ end
+ @form.fields << WWW::Mechanize::Form::Field.new("__EVENTTARGET", target_element ? target_element.name : "") if target_element
+ submit_form_button
+ end
+end
+end
39 lib/web-sickle/lib/helpers/table_reader.rb
@@ -0,0 +1,39 @@
+module WebSickle::Helpers
+class TableReader
+ attr_reader :headers, :options, :body_rows, :header_row, :extra_rows
+
+ def initialize(element, p_options = {})
+ @options = {
+ :row_selectors => [" > tr", "thead > tr", "tbody > tr"],
+ :header_selector => " > th",
+ :header_proc => lambda { |th| th.inner_text.gsub(/[\n\s]+/, ' ').strip },
+ :body_selector => " > td",
+ :body_proc => lambda { |header, td| td.inner_text.strip },
+ :header_offset => 0,
+ :body_offset => 1
+ }.merge(p_options)
+ @options[:body_range] ||= options[:body_offset]..-1
+ raw_rows = options[:row_selectors].map{|row_selector| element / row_selector}.compact.flatten
+
+ @header_row = raw_rows[options[:header_offset]]
+ @body_rows = raw_rows[options[:body_range]]
+ @extra_rows = (options[:body_range].last+1)==0 ? [] : raw_rows[(options[:body_range].last+1)..-1]
+
+ @headers = (@header_row / options[:header_selector]).map(&options[:header_proc])
+ end
+
+ def rows
+ @rows ||= @body_rows.map do |row|
+ hash = {}
+ data_array = (headers).zip(row / options[:body_selector]).each do |column_name, td|
+ hash[column_name] = options[:body_proc].call(column_name, td)
+ end
+ hash
+ end
+ end
+
+ def array_to_hash(data, column_names)
+ column_names.inject({}) {|h,column_name| h[column_name] = data[column_names.index(column_name)]; h }
+ end
+end
+end
15 lib/web-sickle/lib/make_nokigiri_output_useful.rb
@@ -0,0 +1,15 @@
+# Nokogiri::XML::Element.class_eval do
+# def inspect(indent = "")
+# breaker = "\n#{indent}"
+# if children.length == 0
+# %(#{indent}<#{name}#{breaker} #{attributes.map {|k,v| k + '=' + v.inspect} * "#{breaker} "}/>)
+# else
+# %(#{indent}<#{name} #{attributes.map {|k,v| k + '=' + v.inspect} * " "}>\n#{children.map {|c| c.inspect(indent + ' ') rescue c.class} * "\n"}#{breaker}</#{name}>)
+# end
+# end
+# end
+# Nokogiri::XML::Text.class_eval do
+# def inspect(indent = "")
+# "#{indent}#{text.inspect}"
+# end
+# end
224 lib/web-sickle/lib/web_sickle.rb
@@ -0,0 +1,224 @@
+class WebsickleException < Exception; end
+
+module WebSickle
+ # form_value is used to interface with the current select form
+ attr_reader :form_value
+ attr_accessor :page
+
+ def initialize(options = {})
+ @page = nil
+ @form_value = HashProxy.new(
+ :set => lambda { |identifier, value| set_form_value(identifier, value)},
+ :get => lambda { |identifier| get_form_value(identifier)}
+ )
+ end
+
+ def click_link(link)
+ set_page(agent.click(find_link(link)))
+ end
+
+ def submit_form(options = {})
+ options[:button] = :first unless options.has_key?(:button)
+ options[:identified_by] ||= :first
+ select_form(options[:identified_by])
+ set_form_values(options[:values]) if options[:values]
+ submit_form_button(options[:button])
+ end
+
+ # select the current form
+ def select_form(identifier = {})
+ identifier = make_identifier(identifier, [:name, :action, :method])
+ @form = find_in_collection(@page.forms, identifier)
+ report_error("Couldn't find form on page at #{@page.uri} with attributes #{identifier.inspect}") if @form.nil?
+ @form
+ end
+
+ # submits the current form
+ def submit_form_button(button_criteria = nil, options = {})
+ button =
+ case button_criteria
+ when nil
+ nil
+ else
+ find_button(button_criteria)
+ end
+ set_page(agent.submit(@form, button))
+ end
+
+ # sets the given path to the current page, then opens it using our agent
+ def open_page(path, parameters = [], referer = nil)
+ set_page(agent.get(path, parameters, referer))
+ end
+
+ # uses Hpricot style css selectors to find the elements in the current +page+.
+ # Uses Hpricot#/ (or Hpricot#search)
+ def select_element(match)
+ select_element_in(@page, match)
+ end
+
+ # uses Hpricot style css selectors to find the element in the given container. Works with html pages, and file pages that happen to have xml-like content.
+ # throws error if can't find a match
+ def select_element_in(contents, match)
+ result = (contents.respond_to?(:/) ? contents : Hpricot(contents.body)) / match
+ if result.blank?
+ report_error("Tried to find element matching #{match}, but couldn't")
+ else
+ result
+ end
+ end
+
+ # uses Hpricot style css selectors to find the element. Works with html pages, and file pages that happen to have xml-like content.
+ # throws error if can't find a match
+ # Uses Hpricot#at
+ def detect_element(match)
+ result = (@page.respond_to?(:at) ? @page : Hpricot(@page.body)).at(match)
+ if result.blank?
+ report_error("Tried to find element matching #{match}, but couldn't")
+ else
+ result
+ end
+ end
+
+ protected
+ # our friendly mechinze agent
+ def agent
+ @agent ||= new_mechanize_agent
+ end
+
+ def make_identifier(identifier, valid_keys = nil, default_key = :name)
+ identifier = {default_key => identifier} unless identifier.is_a?(Hash) || identifier.is_a?(Symbol)
+ identifier.assert_valid_keys(valid_keys) if identifier.is_a?(Hash) && valid_keys
+ identifier
+ end
+
+ def find_field(identifier)
+ if @form.nil?
+ report_error("No form is selected when trying to find field by #{identifier.inspect}")
+ return
+ end
+ identifier = make_identifier(identifier, [:name, :value])
+ find_in_collection(@form.radiobuttons + @form.fields + @form.checkboxes + @form.file_uploads, identifier) ||
+ report_error("Tried to find field identified by #{identifier.inspect}, but failed.\nForm fields are: #{(@form.radiobuttons + @form.fields + @form.checkboxes + @form.file_uploads).map{|f| f.inspect} * ", \n "}")
+ end
+
+ def find_link(identifier)
+ identifier = make_identifier(identifier, [:href, :text], :text)
+ find_in_collection(page.links, identifier) ||
+ report_error("Tried to find link identified by #{identifier.inspect}, but failed.\nValid links are: #{page.links.map{|f| f.inspect} * ", \n "}")
+ end
+
+ # finds a button by parameters. Throws error if not able to find.
+ # example:
+ # find_button("btnSubmit") - finds a button named "btnSubmit"
+ # find_button(:name => "btnSubmit")
+ # find_button(:name => "btnSubmit", :value => /Lucky/) - finds a button named btnSubmit with a value matching /Lucky/
+ def find_button(identifier)
+ identifier = make_identifier(identifier, [:value, :name])
+ find_in_collection(@form.buttons, identifier) ||
+ report_error("Tried to find button identified by #{identifier.inspect}, but failed. Buttons on selected form are: #{@form.buttons.map{|f| f.inspect} * ','}")
+ end
+
+ # the magic method that powers find_button, find_field. Does not throw an error if not found
+ def find_in_collection(collection, identifier, via = :find)
+ return collection.first if identifier == :first
+ find_all_in_collection(collection, identifier, :find)
+ end
+
+ def find_all_in_collection(collection, identifier, via = :select)
+ return [collection.first] if identifier == :first
+ collection.send(via) do |item|
+ identifier.all? { |k, criteria| is_a_match?(criteria, item.send(k)) }
+ end
+ end
+
+ # sets a form-field's value by identifier. Throw's error if field does not exist
+ def set_form_value(identifier, value)
+ field = find_field(identifier)
+ case field
+ when WWW::Mechanize::Form::CheckBox
+ field.checked = value
+ when WWW::Mechanize::Form::RadioButton
+ radio_collection = find_all_in_collection(@form.radiobuttons, :name => field.name)
+ radio_collection.each { |f|; f.checked = false }
+ finder = (value.is_a?(Hash) || value.is_a?(Symbol)) ? value : {:value => value}
+ find_in_collection(radio_collection, finder).checked = true
+ when WWW::Mechanize::Form::SelectList
+ if value.is_a?(Hash) || value.is_a?(Symbol)
+ field.value = find_in_collection(field.options, value).value
+ else
+ field.value = value
+ end
+ else
+ field.value = value
+ end
+ end
+
+ def set_form_values(set_pairs = {})
+ flattened_value_hash(set_pairs).each do |identifier, value|
+ set_form_value(identifier, value)
+ end
+ end
+
+ def flattened_value_hash(hash, parents = [])
+ new_hash = {}
+ hash.each do |key, value|
+ if value.is_a?(Hash) && value.keys.first.is_a?(String)
+ new_hash.update(flattened_value_hash(value, [key] + parents))
+ else
+ parents.each { |parent| key = "#{parent}[#{key}]"}
+ new_hash[key] = value
+ end
+ end
+ new_hash
+ end
+
+ # sets a form-field's value by identifier. Throw's error if field does not exist
+ def get_form_value(identifier)
+ field = find_field(identifier)
+ case field
+ when WWW::Mechanize::Form::CheckBox
+ field.checked
+ else
+ field.value
+ end
+ end
+
+ def format_error(msg)
+ error = "Error encountered: #{msg}."
+ begin
+ error << "\n\nPage URL:#{@page.uri.to_s}" if @page
+ rescue
+ end
+ error
+ end
+
+ def report_error(msg)
+ raise WebsickleException, format_error(msg)
+ nil
+ end
+
+ private
+ def set_page(p)
+ @form = nil
+ @page = p
+ end
+
+ def is_a_match?(criteria, value)
+ case criteria
+ when Regexp
+ criteria.match(value)
+ when String
+ criteria == value
+ when Array
+ criteria.include?(value)
+ else
+ criteria.to_s == value.to_s
+ end
+ end
+
+ def new_mechanize_agent
+ a = WWW::Mechanize.new
+ a.read_timeout = 600 # 10 minutes
+ a
+ end
+end
9 lib/web-sickle/spec/fixtures/linkies.html
@@ -0,0 +1,9 @@
+<html>
+<head>
+ <title>Boogie</title>
+</head>
+<body>
+ <a href="/link/to/one" id="link_number_one">Link number one</a>
+ <a href="/link/to/two" id="link_number_one">Link number two</a>
+</body>
+</html>
137 lib/web-sickle/spec/lib/helpers/table_reader_spec.rb
@@ -0,0 +1,137 @@
+require File.dirname(__FILE__) + '/../../spec_helper'
+
+describe WebSickle::Helpers::TableReader do
+ describe "Simple example" do
+ before(:each) do
+ @content = <<-EOF
+ <table>
+ <tr>
+ <th>Name</th>
+ <th>Age</th>
+ </tr>
+ <tr>
+ <td>Googly</td>
+ <td>2</td>
+ </tr>
+ </table>
+ EOF
+ h = Hpricot(@content)
+ @table = WebSickle::Helpers::TableReader.new(h / "table")
+ end
+
+ it "should extract headers" do
+ @table.headers.should == ["Name", "Age"]
+ end
+
+ it "should extract rows" do
+ @table.rows.should == [
+ {"Name" => "Googly", "Age" => "2"}
+ ]
+ end
+ end
+
+
+
+ describe "Targetted example" do
+ before(:each) do
+ @content = <<-EOF
+ <table>
+ <thead>
+ <tr>
+ <td colspan='2'>----</td>
+ </tr>
+ <tr>
+ <th><b>Name</b></th>
+ <th><b>Age</b></th>
+ </tr>
+ </thead>
+ <tbody>
+ <tr>
+ <td>Googly</td>
+ <td>2</td>
+ </tr>
+ <tr>
+ <td>Bear</td>
+ <td>3</td>
+ </tr>
+ <tr>
+ <td colspan='2'>Totals!</td>
+ </tr>
+ <tr>
+ <td>---</td>
+ <td>5</td>
+ </tr>
+ </tbody>
+ </table>
+ EOF
+ h = Hpricot(@content)
+ @table = WebSickle::Helpers::TableReader.new(h / " > table",
+ :header_selector => " > th > b",
+ :header_offset => 1,
+ :body_range => 2..-3
+ )
+ end
+
+ it "should extract the column headers" do
+ @table.headers.should == ["Name", "Age"]
+ end
+
+ it "should extract the row data for the specified range" do
+ @table.rows.should == [
+ {"Name" => "Googly", "Age" => "2"},
+ {"Name" => "Bear", "Age" => "3"},
+ ]
+ end
+
+ it "should allow you to check extra rows to assert you didn't chop off too much" do
+ (@table.extra_rows.first / "td").inner_text.should == "Totals!"
+ end
+ end
+
+
+
+ describe "when using procs to extract data" do
+ before(:each) do
+ @content = <<-EOF
+ <table>
+ <tr>
+ <th>Name</th>
+ <th>Age</th>
+ </tr>
+ <tr>
+ <td>Googly</td>
+ <td>2</td>
+ </tr>
+ <tr>
+ <td>Bear</td>
+ <td>3</td>
+ </tr>
+ </table>
+ EOF
+ h = Hpricot(@content)
+ @table = WebSickle::Helpers::TableReader.new(h / " > table",
+ :header_proc => lambda {|th| th.inner_text.downcase.to_sym},
+ :body_proc => lambda {|col_name, td|
+ value = td.inner_text
+ case col_name
+ when :name
+ value.upcase
+ when :age
+ value.to_i
+ end
+ }
+ )
+ end
+
+ it "should use the header proc to extract column headers" do
+ @table.headers.should == [:name, :age]
+ end
+
+ it "should use the body proc to format the data" do
+ @table.rows.should == [
+ {:name => "GOOGLY", :age => 2},
+ {:name => "BEAR", :age => 3}
+ ]
+ end
+ end
+end
7 lib/web-sickle/spec/spec_helper.rb
@@ -0,0 +1,7 @@
+require File.dirname(__FILE__) + '/../init.rb'
+require 'rubygems'
+require 'hpricot'
+require 'test/unit'
+require 'spec'
+require 'active_support'
+require File.dirname(__FILE__) + '/spec_helpers/mechanize_mock_helper.rb'
12 lib/web-sickle/spec/spec_helpers/mechanize_mock_helper.rb
@@ -0,0 +1,12 @@
+module MechanizeMockHelper
+ def fixture_file(filename)
+ File.read("#{File.dirname(__FILE__)}/../fixtures/#{filename}")
+ end
+
+ def mechanize_page(path_to_data, options = {})
+ options[:uri] ||= URI.parse("http://url.com/#{path_to_data}")
+ options[:response] ||= {'content-type' => 'text/html'}
+
+ WWW::Mechanize::Page.new(options[:uri], options[:response], fixture_file("/#{path_to_data}"))
+ end
+end
50 lib/web-sickle/spec/web_sickle_spec.rb
@@ -0,0 +1,50 @@
+require File.dirname(__FILE__) + '/spec_helper'
+
+class WebSickleHelper
+ include WebSickle
+end
+
+describe WebSickle do
+ include MechanizeMockHelper
+
+ before(:all) do
+ WebSickleHelper.protected_instance_methods.each do |method|
+ WebSickleHelper.send(:public, method)
+ end
+ end
+
+ before(:each) do
+ @helper = WebSickleHelper.new
+ end
+
+ it "should flatten a value hash" do
+ @helper.flattened_value_hash("contact" => {"first_name" => "bob"}).should == {"contact[first_name]" => "bob"}
+ end
+
+ describe "clicking links" do
+ before(:each) do
+ @helper.stub!(:page).and_return(mechanize_page("linkies.html"))
+ end
+
+ it "should click a link by matching the link text" do
+ @helper.agent.should_receive(:click) do |link|
+ link.text.should include("one")
+ end
+ @helper.click_link(:text => /one/)
+ end
+
+ it "should click a link by matching the link href" do
+ @helper.agent.should_receive(:click) do |link|
+ link.href.should include("/two")
+ end
+ @helper.click_link(:href => %r{/two})
+ end
+
+ it "should default matching the link text" do
+ @helper.agent.should_receive(:click) do |link|
+ link.text.should include("Link number one")
+ end
+ @helper.click_link("Link number one")
+ end
+ end
+end

0 comments on commit 8b21619

Please sign in to comment.
Something went wrong with that request. Please try again.