Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Interested in merging features? #21

Closed
wants to merge 31 commits into from

3 participants

@invernizzi

Hello Relevance people,
nashby has added rails3 support for Tarantula (I'm using it, it works fine).
I've fixed the TidyHandler to work on Os X and added a "blessing" feature, to mark errors as false positives/stuff I don't care about or want to fix.
Are you interested in pulling these? If so, I will split them in separate merges (otherwise is useless work).
Bye!

nashby and others added some commits
@nashby nashby added rails 3 support 6f756f0
@nashby nashby fixed all examples except mutate_selects 756baff
@invernizzi invernizzi the html? method always returned the same value 0015042
@invernizzi invernizzi Explaining why tarantula raises an exception upon failure 11ae49a
@invernizzi invernizzi Tidy works on linux and osX without the need of env variables 05ebd10
@invernizzi invernizzi other fixes to TidyHandler 9ef6dd7
@invernizzi invernizzi rails3 is fussing with git :) b963d6f
@invernizzi invernizzi typo cf88a8d
@invernizzi invernizzi Ignoring errors is now possible (and, hopefully, very easy).
The Html template explains to the user how to do it.
The CLI version doesn't show it currently.
bbbfc86
@invernizzi invernizzi Handlers can report multiple results. Tidy does so 39ab5dd
@invernizzi invernizzi Dependency to tidy added b2bd6f5
@invernizzi invernizzi Adding an initial set of XSS checks. Verification of their effectiven…
…ess is

still a TODO.
f5c1f73
@invernizzi invernizzi removing the automatic mixin wit IntegrationTest, since is needed onl…
…y for

Tarantula's tests. Added it to the test template instead.
d3f00bc
@invernizzi invernizzi Gemspec updated 8c2d3b2
@invernizzi invernizzi Following freedesktop.org standards 983fc04
@invernizzi invernizzi Version bump to 0.4.0 e8d5b44
@invernizzi invernizzi jeweler updated bf6a81e
@invernizzi invernizzi Regenerate gemspec for version 0.4.0 8d1fea7
@invernizzi invernizzi Tidy: ignore proprietary attributes f967d46
@invernizzi invernizzi Handle the case where tidy is not installed 3bfe859
@invernizzi invernizzi Removed exception on tests failure bc8c9e6
@invernizzi invernizzi Handle the case where tidy is not installed 7164ef0
@invernizzi invernizzi Merge branch 'master' of github.com:invernizzi/tarantula-rails3 a6f08d8
@invernizzi invernizzi reverting last commit 029646c
@invernizzi invernizzi the crawler now remembers the problems found, for later inspection bd0d169
@invernizzi invernizzi feature: checking that generic text (javascript etc) is present in th…
…e rendered pages
5ea8d13
@invernizzi invernizzi Better explanation of invalid uris 177b83d
@invernizzi invernizzi correct url encoding 126e838
@invernizzi invernizzi stable blessing of XSS/SQL injections 6436fc5
@invernizzi invernizzi Adding commit hooks that build the gem 26696c2
@invernizzi invernizzi modifing commit hooks to build gem 24c13d5
@jgkite
Owner

Hi

We may be interested in these, but we would need smaller, separate pull requests based on the latest HEAD. Also note that rails 3 work has been merged in as well.

Thanks, @rsanheim and Jamie

@jgkite jgkite closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 14, 2011
  1. @nashby

    added rails 3 support

    nashby authored
Commits on Feb 18, 2011
  1. @nashby
Commits on Jun 21, 2011
  1. @invernizzi
  2. @invernizzi
  3. @invernizzi
Commits on Jun 22, 2011
  1. @invernizzi

    other fixes to TidyHandler

    invernizzi authored
  2. @invernizzi
  3. @invernizzi

    typo

    invernizzi authored
Commits on Jun 24, 2011
  1. @invernizzi

    Ignoring errors is now possible (and, hopefully, very easy).

    invernizzi authored invernizzi committed
    The Html template explains to the user how to do it.
    The CLI version doesn't show it currently.
  2. @invernizzi
Commits on Jun 27, 2011
  1. @invernizzi

    Dependency to tidy added

    invernizzi authored
  2. @invernizzi
  3. @invernizzi

    removing the automatic mixin wit IntegrationTest, since is needed onl…

    invernizzi authored
    …y for
    
    Tarantula's tests. Added it to the test template instead.
Commits on Jun 28, 2011
  1. @invernizzi

    Gemspec updated

    invernizzi authored
  2. @invernizzi

    Following freedesktop.org standards

    invernizzi authored invernizzi committed
  3. @invernizzi

    Version bump to 0.4.0

    invernizzi authored
  4. @invernizzi

    jeweler updated

    invernizzi authored
  5. @invernizzi
Commits on Jul 21, 2011
  1. @invernizzi
  2. @invernizzi
Commits on Jul 25, 2011
  1. @invernizzi
  2. @invernizzi
  3. @invernizzi
  4. @invernizzi

    reverting last commit

    invernizzi authored
Commits on Jul 26, 2011
  1. @invernizzi
Commits on Aug 17, 2011
  1. @invernizzi
  2. @invernizzi
  3. @invernizzi

    correct url encoding

    invernizzi authored
Commits on Aug 18, 2011
  1. @invernizzi
Commits on Aug 19, 2011
  1. @invernizzi
  2. @invernizzi
This page is out of date. Refresh to see the latest.
Showing with 592 additions and 107 deletions.
  1. +0 −1  .gitignore
  2. +3 −27 README.rdoc
  3. +9 −26 Rakefile
  4. +3 −2 VERSION.yml
  5. +1 −1  examples/relevance/core_extensions/file_example.rb
  6. +1 −1  examples/relevance/tarantula/form_submission_example.rb
  7. +4 −4 examples/relevance/tarantula/link_example.rb
  8. +3 −0  git-hooks/pre-commit
  9. +1 −1  lib/relevance/core_extensions/response.rb
  10. +189 −7 lib/relevance/core_extensions/test_case.rb
  11. +4 −3 lib/relevance/tarantula.rb
  12. +1 −0  lib/relevance/tarantula/attack_handler.rb
  13. +44 −0 lib/relevance/tarantula/blessing.rb
  14. +31 −8 lib/relevance/tarantula/crawler.rb
  15. +21 −0 lib/relevance/tarantula/detail.html.erb
  16. +1 −1  lib/relevance/tarantula/form_submission.rb
  17. +1 −0  lib/relevance/tarantula/html_report_helper.rb
  18. +18 −0 lib/relevance/tarantula/link.rb
  19. +7 −1 lib/relevance/tarantula/rails_integration_proxy.rb
  20. +1 −1  lib/relevance/tarantula/result.rb
  21. +24 −0 lib/relevance/tarantula/text_presence.rb
  22. +32 −6 lib/relevance/tarantula/tidy_handler.rb
  23. +9 −6 { → lib/relevance}/tasks/tarantula_tasks.rake
  24. +9 −0 lib/tarantula-rails3.rb
  25. BIN  pkg/tarantula-rails3-0.4.0.gem
  26. +144 −0 tarantula-rails3.gemspec
  27. +14 −10 tarantula.gemspec
  28. +14 −0 template/blessed.rb
  29. +3 −1 template/tarantula_test.rb
View
1  .gitignore
@@ -1,6 +1,5 @@
rcov_tmp
tmp
-pkg
rdoc
.zsh_rake_cache
doc
View
30 README.rdoc
@@ -2,7 +2,7 @@
== DESCRIPTION
-Tarantula is a big fuzzy spider. It crawls your Rails application, fuzzing data to see what breaks.
+Tarantula is a big fuzzy spider. It crawls your Rails 3 application, fuzzing data to see what breaks.
== Usage
@@ -11,32 +11,7 @@ Tarantula is a big fuzzy spider. It crawls your Rails application, fuzzing data
The latest and greatest version is always available on GitHub. (See the rakefile for dependencies, or
just let RubyGems handle it.)
- gem install tarantula
-
-You can also grab it from RubyForge, where we will push stable releases but may not be as bleeding edge
-as the GitHub gem.
-
- gem install tarantula
-
-=== Project Setup
-
-To set up Tarantula into your application, add the following line into either config/environment.rb or
-config/environments/test.rb (preferred). This assumes that you have Rails 2.1 or higher installed.
-
- config.gem 'tarantula', :lib => 'relevance/tarantula'
-
-Since Rails doesn't (yet) support automatically loading rake tasks that live inside gems, you will need
-to update your Rakefile to load Tarantula's rake tasks. The simplest approach is to start by vendoring
-Tarantula into your Rails app.
-
- mkdir -p vendor/gems
- cd vendor/gems
- gem unpack tarantula
-
-You can then add the following line into your Rakefile, which will allow your application to discover
-Tarantula's rake tasks.
-
- load File.join(RAILS_ROOT, Dir["vendor/gems/tarantula-*/tasks/*.rake"])
+ gem install tarantula-rails3
=== Crawling Your App
@@ -50,6 +25,7 @@ Take a moment to familiarize yourself with the generated test. If parts of your
login, update the test to make sure Tarantula can access those parts of your app.
require "relevance/tarantula"
+ require "test_helper"
class TarantulaTest < ActionController::IntegrationTest
# Load enough test data to ensure that there's a link to every page in your
View
35 Rakefile
@@ -1,25 +1,27 @@
require 'rake'
require 'rake/testtask'
require 'rake/rdoctask'
+$LOAD_PATH.unshift(File.expand_path(File.dirname(__FILE__) + '/../micronaut/lib/'))
require 'micronaut'
require 'micronaut/rake_task'
begin
require 'jeweler'
- files = ["CHANGELOG", "MIT-LICENSE", "Rakefile", "README.rdoc", "VERSION.yml"]
+ files = ["CHANGELOG", "LICENSE", "Rakefile", "README.rdoc", "VERSION.yml"]
files << Dir["examples/**/*", "laf/**/*", "lib/**/*", "tasks/**/*", "template/**/*"]
Jeweler::Tasks.new do |s|
- s.name = "tarantula"
+ s.name = "tarantula-rails3"
s.summary = "A big hairy fuzzy spider that crawls your site, wreaking havoc"
s.description = "A big hairy fuzzy spider that crawls your site, wreaking havoc"
- s.homepage = "http://github.com/relevance/tarantula"
- s.email = "opensource@thinkrelevance.com"
- s.authors = ["Relevance, Inc."]
+ s.homepage = "http://github.com/invernizzi/tarantula-rails3"
+ s.email = ["opensource@thinkrelevance.com", "invernizzi.l@gmail.com"]
+ s.authors = ["Relevance, Inc.", "Luca Invernizzi"]
s.require_paths = ["lib"]
s.files = files.flatten
s.add_dependency 'htmlentities', '>= 4.2.0'
s.add_dependency 'hpricot', '>= 0.8.1'
+ s.add_dependency "tidy"
s.add_development_dependency 'micronaut'
s.add_development_dependency 'log_buddy'
end
@@ -30,35 +32,16 @@ end
desc "Run all micronaut examples"
Micronaut::RakeTask.new :examples do |t|
- t.pattern = "examples/**/*_example.rb"
+ t.pattern = File.dirname(__FILE__)+"/examples/**/*_example.rb"
end
desc "Run all micronaut examples using rcov"
Micronaut::RakeTask.new :rcov do |t|
- t.pattern = "examples/**/*_example.rb"
+ t.pattern = File.dirname(__FILE__)+"/examples/**/*_example.rb"
t.rcov = true
t.rcov_opts = %[--exclude "gems/*,/Library/Ruby/*,config/*" --text-summary --sort coverage]
end
-namespace :examples do
-
- RAILS_VERSIONS = %w[2.3.2 2.3.4]
-
- unless RUBY_VERSION =~ /^1\.9\./
- RAILS_VERSIONS.unshift(*%w[2.0.2 2.1.0 2.1.1 2.2.2 2.3.3])
- RAILS_VERSIONS.sort!
- end
-
- desc "Run examples with multiple versions of rails"
- task :multi_rails do
- RAILS_VERSIONS.each do |rails_version|
- puts
- sh "RAILS_VERSION='#{rails_version}' rake examples"
- end
- end
-
-end
-
if ENV["RUN_CODE_RUN"]
task :default => [:check_dependencies, "examples:multi_rails"]
else
View
5 VERSION.yml
@@ -1,4 +1,5 @@
---
:major: 0
-:minor: 3
-:patch: 3
+:minor: 4
+:build:
+:patch: 0
View
2  examples/relevance/core_extensions/file_example.rb
@@ -5,4 +5,4 @@
it "should return the extension without the leading dot" do
File.extension("foo.bar").should == "bar"
end
-end
+end
View
2  examples/relevance/tarantula/form_submission_example.rb
@@ -53,7 +53,7 @@
end
it "can mutate selects" do
- Hpricot::Elements.any_instance.stubs(:rand).returns(stub(:[] => "2006-stub"))
+ @fs.attack.stubs(:rand).returns(stub(:[] => "2006-stub"))
@fs.mutate_selects(@form).should == {"foo[opened_on(1i)]" => "2006-stub"}
end
View
8 examples/relevance/tarantula/link_example.rb
@@ -16,19 +16,19 @@
end
it "parses anchor tags with POST 'method'" do
- link = make_link(Hpricot(%Q{<a href="/foo" onclick="#{method_javascript_function(:post)}">foo</a>}).at('a'))
+ link = make_link(Hpricot(%Q{<a href="/foo" onclick="#{Relevance::Tarantula::Link.method_javascript_function(:post)}">foo</a>}).at('a'))
link.href.should == '/foo'
link.method.should == :post
end
it "parses anchor tags with PUT 'method'" do
- link = make_link(Hpricot(%Q{<a href="/foo" onclick="#{method_javascript_function(:put)}">foo</a>}).at('a'))
+ link = make_link(Hpricot(%Q{<a href="/foo" onclick="#{Relevance::Tarantula::Link.method_javascript_function(:put)}">foo</a>}).at('a'))
link.href.should == '/foo'
link.method.should == :put
end
it "parses anchor tags with DELETE 'method'" do
- link = make_link(Hpricot(%Q{<a href="/foo" onclick="#{method_javascript_function(:delete)}">foo</a>}).at('a'))
+ link = make_link(Hpricot(%Q{<a href="/foo" onclick="#{Relevance::Tarantula::Link.method_javascript_function(:delete)}">foo</a>}).at('a'))
link.href.should == '/foo'
link.method.should == :delete
end
@@ -81,4 +81,4 @@ class Link < ActiveRecord::Base
}.should_not raise_error
end
-end
+end
View
3  git-hooks/pre-commit
@@ -0,0 +1,3 @@
+#!/bin/sh
+rake gemspec:generate build
+git add pkg/*.gem
View
2  lib/relevance/core_extensions/response.rb
@@ -3,7 +3,7 @@ module Relevance::CoreExtensions::Response
def html?
# some versions of Rails integration tests don't set content type
# so we are treating nil as html. A better fix would be welcome here.
- ((content_type =~ %r{^text/html}) != nil) || content_type == nil
+ ((content_type.to_s =~ %r{^text/html.*}) != nil) || content_type == nil
end
end
View
196 lib/relevance/core_extensions/test_case.rb
@@ -1,4 +1,4 @@
-require 'action_controller/integration'
+require 'action_dispatch/testing/integration'
module Relevance::CoreExtensions::TestCaseExtensions
@@ -7,13 +7,195 @@ def tarantula_crawl(integration_test, options = {})
t = tarantula_crawler(integration_test, options)
t.crawl url
end
-
+
def tarantula_crawler(integration_test, options = {})
- Relevance::Tarantula::RailsIntegrationProxy.rails_integration_test(integration_test, options)
+ Relevance::Tarantula::RailsIntegrationProxy.rails_integration_test(integration_test, options)
+ end
+
+ def tarantula_activate_HTML_checking(crawler)
+ crawler.handlers << Relevance::Tarantula::TidyHandler.new
end
-
+
+ def tarantula_activate_form_submissions_attacks(crawler)
+ #trivial XSS
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => "<script>bad</script>",
+ :output => "<script>bad</script>",
+ }
+
+ #Closing tag
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => "><script>bad</script>&",
+ :output => "<script>bad</script>",
+ }
+
+ #style tag
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => '<style>@import"javascript:bad";</style>',
+ :output => '<style>@import"javascript:bad";</style>',
+ }
+
+ #check escaping
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => ''';!--"<XSS>=&{()}',
+ :output => "<XSS", #correct escaping would be &lt;XSS
+ }
+
+ #javascript in images
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => '<IMG SRC=javascript:bad>',
+ :output => '<IMG SRC=javascript:bad>',
+ }
+
+ #javascript in images, embedded tab
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => "<IMG SRC=java\tscript:bad>",
+ :output => "<IMG SRC=java\tscript:bad>",
+ }
+
+ #javascript in images, embedded tab, encoded
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => "<IMG SRC=java&#x09script:bad>",
+ :output => "<IMG SRC=java\tscript:bad>",
+ }
+
+ #javascript in images, embedded newline
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => "<IMG SRC=java\nscript:bad>",
+ :output => "<IMG SRC=java\nscript:bad>",
+ }
+
+ #javascript in images, embedded newline, encoded
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => "<IMG SRC=java&#x0Ascript:bad>",
+ :output => "<IMG SRC=java\nscript:bad>",
+ }
+
+ #javascript in images, embedded carriage return
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => "<IMG SRC=java\rscript:bad>",
+ :output => "<IMG SRC=java\rscript:bad>",
+ }
+
+ #javascript in images, embedded carriage return, encoded
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => "<IMG SRC=java&#x0Dscript:bad>",
+ :output => "<IMG SRC=java\rscript:bad>",
+ }
+
+ #javascript in images, embedded null
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => "<IM\0G SRC=java\0script:bad>",
+ :output => "<IM\0G SRC=java\0script:bad>",
+ }
+
+ #javascript in images, embedded newline, multiline
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => "<IMG\n SRC\n=\njava\nscript\n:\nbad>",
+ :output => "<IMG\n SRC\n=\njava\nscript\n:\nbad>",
+ }
+
+ #malformed tags
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => '<IMG """><SCRIPT>bad</SCRIPT>">',
+ :output => '<SCRIPT>bad</SCRIPT>',
+ }
+
+ #UTF-8 encoding in hex
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => '<IMG src=&#x6A;&#x61;&#x76;&#x61;&#x73;&#x63;&#x72;&#x69;&#x70;&#x74;&#x3A;&#x62;&#x61;&#x64;>',
+ :output => '<IMG src=javascript:bad>',
+ }
+
+ #UTF-8 encoding in hex, long strings
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => '<IMG src=&#x000006A;&#x0000061;&#x0000076;&#x0000061;&#x0000073;&#x0000063;&#x0000072;&#x0000069;&#x0000070;&#x0000074;&#x000003A;&#x0000062;&#x0000061;&#x0000064;>',
+ :output => '<IMG src=javascript:bad>',
+ }
+
+ #UTF-8 encoding in decimal
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => '<IMG src=&#106&#97&#118&#97&#115&#99&#114&#105&#112&#116&#58&#98&#97&#100>',
+ :output => '<IMG src=javascript:bad>',
+ }
+
+
+ #Ruby's #{}. It's probably impossible, but it doesn't hurt to check
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => '#{351*887}',
+ :output => "311337",
+ }
+
+ #Non-alpha-non-digit XSS
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => '<script/ bad',
+ :output => "<script> bad",
+ }
+
+ #Extraneous open brackets
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => '<<script>bad;//<</script>',
+ :output => "<script>bad",
+ }
+
+ #unclosed tag
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => '<script>bad',
+ :output => "<script>bad",
+ }
+
+ #escaping javascript
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => '\";<script>bad',
+ :output => "<script>bad",
+ }
+
+ #in title
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => '</title><script>bad</script>',
+ :output => "<script>bad",
+ }
+
+ #possible encodings of "<" (not all of them, they are too many)
+ ['<', '%3C', '&lt', '&lt;', '&LT', '&LT;', '&#60', '&#060', '&#0000060',
+ '&#60;', '&#060;', '&#0000060;', '&#x3c', '&#x03c', '&#x000003c',
+ '&#x3c;', '&#x03c;', '&#x000003c;', '&#X3c', '&#X03c', '&#X000003c',
+ '&#X3c;', '&#X03c;', '&#X000003c;', '&#x3C', '&#x03C', '&#x000003C',
+ '&#x3C;', '&#x03C;', '&#x003C;', '&#x000003C;',
+ '&#X3C', '&#X000003C', '&#X3C;', '&#X000003C;', '\x3c',
+ '\x3C', '\u003c', '\u003C'].each do |encoding|
+ Relevance::Tarantula::FormSubmission.attacks << {
+ :name => :xss,
+ :input => encoding + "0598729",
+ :output => "<" + "0598729",
+ }
+ end
+
+ crawler.handlers << Relevance::Tarantula::AttackHandler.new
+ end
+
end
-if defined? ActionController::IntegrationTest
- ActionController::IntegrationTest.class_eval { include Relevance::CoreExtensions::TestCaseExtensions }
-end
View
7 lib/relevance/tarantula.rb
@@ -4,7 +4,6 @@
require 'erb'
require 'active_support'
require 'action_controller'
-
# bringing in xss-shield requires a bunch of other dependencies
# still not certain about this, if it ruins your world please let me know
#xss_shield_path = File.join(TARANTULA_ROOT, %w{vendor xss-shield})
@@ -24,7 +23,7 @@ def log(msg)
puts msg if verbose
end
def rails_root
- ::RAILS_ROOT
+ ::Rails.root.to_s
end
def verbose
ENV["VERBOSE"]
@@ -55,5 +54,7 @@ def verbose
require File.expand_path(File.join(File.dirname(__FILE__), "tarantula", "attack"))
require File.expand_path(File.join(File.dirname(__FILE__), "tarantula", "attack_handler"))
require File.expand_path(File.join(File.dirname(__FILE__), "tarantula", "link"))
+require File.expand_path(File.join(File.dirname(__FILE__), "tarantula", "blessing"))
-require File.expand_path(File.join(File.dirname(__FILE__), "tarantula", "tidy_handler")) if ENV['TIDY_PATH']
+require File.expand_path(File.join(File.dirname(__FILE__), "tarantula", "tidy_handler"))
+require File.expand_path(File.join(File.dirname(__FILE__), "tarantula", "text_presence"))
View
1  lib/relevance/tarantula/attack_handler.rb
@@ -31,6 +31,7 @@ def handle(result)
########################################################################
#{attacks.select {|a| a.output == $1}[0].to_yaml}
STR
+ error_result.data_blessing = attacks.select {|a| a.output == $1}[0].to_yaml
error_result
end
end
View
44 lib/relevance/tarantula/blessing.rb
@@ -0,0 +1,44 @@
+module Relevance::Tarantula::Blessing
+
+ require 'digest/md5'
+
+ @@BLESSING_FILE = ::Rails.root.to_s + '/test/tarantula/blessed.rb'
+
+ #load the user-defined blessings
+ begin
+ require "#{@@BLESSING_FILE}"
+ blessings = Blessings.blessings
+ rescue LoadError => e
+ puts "Unable to load blessings file (#{@@BLESSING_FILE}), continuing without it. Exception: #{e.message}"
+ blessings = []
+ end
+ #put the blessings into an Hash of hash :=> list of urls regexps
+ @@blessings_hash = Hash.new {|h, k| h[k] = Array.new}
+ blessings.each do |url, hash|
+ url = /^#{Regexp.quote(url)}$/ if url.class == String
+ @@blessings_hash[hash] << url
+ end
+
+ def self.is_blessed?(result)
+ hash = self.hash_result result
+ urls_regexp_array = @@blessings_hash[hash]
+ return false unless urls_regexp_array
+
+ urls_regexp_array.each do |regexp|
+ if result.url =~ regexp
+ return true
+ end
+ end
+ false
+ end
+
+ def self.hash_result (result)
+ Digest::MD5.hexdigest("#{result.description}\n#{result.data_blessing || result.data}\n")
+ end
+
+ def self.get_blessings_path
+ @@BLESSING_FILE
+ end
+
+
+end
View
39 lib/relevance/tarantula/crawler.rb
@@ -54,6 +54,7 @@ def transform_url_patterns=(patterns)
end
def crawl(url = "/")
+ @errors = []
orig_links_queued = @links_queued.dup
orig_form_signatures_queued = @form_signatures_queued.dup
orig_crawl_queue = @crawl_queue.dup
@@ -67,7 +68,7 @@ def crawl(url = "/")
puts e.message
end
- puts "#{(num+1).ordinalize} crawl" if @times_to_crawl > 1
+ puts "#{ActiveSupport::Inflector.ordinalize((num+1))} crawl" if @times_to_crawl > 1
if num + 1 < @times_to_crawl
@links_queued = orig_links_queued
@@ -79,7 +80,12 @@ def crawl(url = "/")
rescue Interrupt
$stderr.puts "CTRL-C"
ensure
- report_results
+ #return the number of problems found
+ @errors = report_results
+ end
+
+ def problems_found?
+ not @errors.empty?
end
def finished?
@@ -101,9 +107,27 @@ def crawl_the_queue(number = 0)
end
end
- def save_result(result)
- reporters.each do |reporter|
- reporter.report(result)
+ def save_result(results)
+ begin
+ #Handlers can return an array of multiple results, or a single result.
+ # Make them all a list
+ unless results.respond_to?("each")
+ results = [results]
+ end
+
+
+ results.each do |result|
+ #if an error is set to be ignored, do so.
+ if result and not result.success
+ result.success = Relevance::Tarantula::Blessing.is_blessed?(result)
+ end
+ reporters.each do |reporter|
+ reporter.report(result)
+ end
+ end
+ rescue Exception => e
+ puts e.message
+ raise
end
end
@@ -113,6 +137,7 @@ def handle_link_results(link, result)
save_result h.handle(result)
rescue Exception => e
log "error handling #{link} #{e.message}"
+ puts e.message
# TODO: pass to results
end
end
@@ -217,9 +242,7 @@ def generate_reports
errors << e
end
end
- unless errors.empty?
- raise errors.map(&:message).join("\n")
- end
+ errors
end
def report_results
View
21 lib/relevance/tarantula/detail.html.erb
@@ -45,6 +45,27 @@
</tbody>
</table>
+ <% unless result.success %>
+ <table class="output">
+ <tbody>
+ <tr>
+ <th colspan="2">#&nbsp;&nbsp;Silence!</th>
+ </tr>
+ <tr>
+ <td colspan="2">
+ To suppress this warning for this URL, add this to the array in
+ <%= Relevance::Tarantula::Blessing.get_blessings_path%>
+ </td>
+ </tr>
+ <tr>
+ <td colspan="2">
+ <%= "['#{result.url}', '#{Relevance::Tarantula::Blessing.hash_result(result)}']," %>
+ </td>
+ </tr>
+ </tbody>
+ </table>
+ <%end %>
+
<table class="output" id="fragment-1">
<tbody>
<tr>
View
2  lib/relevance/tarantula/form_submission.rb
@@ -72,7 +72,7 @@ def mutate_text_areas(form)
def mutate_selects(form)
form.search('select').inject({}) do |form_args, select|
options = select.search('option')
- option = options.rand
+ option = options.sample
form_args[select['name']] = option['value']
form_args
end
View
1  lib/relevance/tarantula/html_report_helper.rb
@@ -1,4 +1,5 @@
require "erb"
+require "active_support/builder" unless defined?(Builder)
module Relevance::Tarantula::HtmlReportHelper
include ERB::Util
include Relevance::Tarantula
View
18 lib/relevance/tarantula/link.rb
@@ -7,6 +7,24 @@ class << self
def protect_against_forgery?
false
end
+ #fast fix for rails3
+ def method_javascript_function(method, url = '', href = nil)
+ action = (href && url.size > 0) ? "'#{url}'" : 'this.href'
+ submit_function =
+ "var f = document.createElement('form'); f.style.display = 'none'; " +
+ "this.parentNode.appendChild(f); f.method = 'POST'; f.action = #{action};"
+
+ unless method == :post
+ submit_function << "var m = document.createElement('input'); m.setAttribute('type', 'hidden'); "
+ submit_function << "m.setAttribute('name', '_method'); m.setAttribute('value', '#{method}'); f.appendChild(m);"
+ end
+
+ if protect_against_forgery?
+ submit_function << "var s = document.createElement('input'); s.setAttribute('type', 'hidden'); "
+ submit_function << "s.setAttribute('name', '#{request_forgery_protection_token}'); s.setAttribute('value', '#{escape_javascript form_authenticity_token}'); f.appendChild(s);"
+ end
+ submit_function << "f.submit();"
+ end
end
METHOD_REGEXPS = {}
View
8 lib/relevance/tarantula/rails_integration_proxy.rb
@@ -30,7 +30,13 @@ def initialize(integration_test)
[:get, :post, :put, :delete].each do |verb|
define_method(verb) do |url, *args|
- integration_test.send(verb, url, *args)
+ begin
+ url = URI.escape url
+ integration_test.send(verb, url, *args)
+ rescue URI::InvalidURIError => e
+ puts "Invalid URI found: url='#{url.inspect}' args='#{args.inspect}'"
+ raise e
+ end
response = integration_test.response
patch_response(url, response)
response
View
2  lib/relevance/tarantula/result.rb
@@ -1,5 +1,5 @@
class Relevance::Tarantula::Result
- HASHABLE_ATTRS = [:success, :method, :url, :response, :referrer, :data, :description, :log, :test_name]
+ HASHABLE_ATTRS = [:success, :method, :url, :response, :referrer, :data, :data_blessing, :description, :log, :test_name]
DEFAULT_LOCALHOST = "http://localhost:3000"
attr_accessor *HASHABLE_ATTRS
include Relevance::Tarantula
View
24 lib/relevance/tarantula/text_presence.rb
@@ -0,0 +1,24 @@
+class Relevance::Tarantula::TextPresence
+ include Relevance::Tarantula
+
+ def initialize(text, error_name, error_description="", ignore_http_codes=[])
+ @text = text
+ @error_name = error_name
+ @error_description = error_description
+ @ignore_http_codes = ignore_http_codes
+ end
+
+ def handle(result)
+ response = result.response
+ error_results = []
+ if response.content_type.to_s == "text/html" and
+ not @ignore_http_codes.include? response.status and
+ not response.body =~ /#{Regexp.escape @text}/
+ error_result = result.dup
+ error_result.description = @error_name
+ error_result.data = @error_description
+ error_results << error_result
+ end
+ error_results
+ end
+end
View
38 lib/relevance/tarantula/tidy_handler.rb
@@ -7,14 +7,31 @@
end
if defined? Tidy
- Tidy.path = ENV['TIDY_PATH'] if ENV['TIDY_PATH']
+ if ENV['TIDY_PATH']
+ #we trust the user
+ Tidy.path = ENV['TIDY_PATH']
+ else
+ begin
+ #linux
+ Tidy.path = '/usr/lib/libtidy.so'
+ rescue LoadError
+ #I'm a mac
+ begin
+ Tidy.path = '/usr/lib/libtidy.dylib'
+ rescue LoadError
+ puts "Tidy library not found, install it"
+ end
+ end
+ end
class Relevance::Tarantula::TidyHandler
include Relevance::Tarantula
def initialize(options = {})
- @options = {:show_warnings=>true}.merge(options)
+ @options = {:show_warnings=>true,
+ :char_encoding=>'utf8'}.merge(options)
end
def handle(result)
+ return unless Tidy.path
response = result.response
return unless response.html?
tidy = Tidy.open(@options) do |tidy|
@@ -22,10 +39,19 @@ def handle(result)
tidy
end
unless tidy.errors.blank?
- error_result = result.dup
- error_result.description = "Bad HTML (Tidy)"
- error_result.data = tidy.errors.inspect
- error_result
+ error_results = []
+ tidy.errors[0].split(/\n/).each do |error|
+ # ignore proprietary attributes
+ next if error.include? "proprietary"
+ error_result = result.dup
+ error_result.description = "Bad HTML (Tidy)"
+ error_result.data = error.inspect
+ # removing the line and column information to let developers suppress a
+ # warning with the Blessing module
+ error_result.data_blessing = error_result.data.gsub(/line\s+\d+\s+column\s+\d+/i, "")
+ error_results << error_result
+ end
+ error_results
end
end
end
View
15 tasks/tarantula_tasks.rake → lib/relevance/tasks/tarantula_tasks.rake
@@ -10,7 +10,7 @@ namespace :tarantula do
t.pattern = 'test/tarantula/**/*_test.rb'
t.verbose = true
end
-
+
Rake::Task[:tarantula_test].invoke
end
@@ -20,13 +20,13 @@ namespace :tarantula do
Rake::Task['tarantula:test'].invoke
rescue RuntimeError => e
puts e.message
- end
-
+ end
+
Dir.glob("tmp/tarantula/**/index.html") do |file|
if PLATFORM['darwin']
system("open #{file}")
elsif PLATFORM[/linux/]
- system("firefox #{file}")
+ system("xdg-open #{file}")
else
puts "You can view tarantula results at #{file}"
end
@@ -36,7 +36,10 @@ namespace :tarantula do
desc 'Generate a default tarantula test'
task :setup do
mkdir_p "test/tarantula"
- template_path = File.expand_path(File.join(File.dirname(__FILE__), "..", "template", "tarantula_test.rb"))
- cp template_path, "test/tarantula/"
+ templates_path = File.expand_path(File.join(File.dirname(__FILE__), "../../..", "template"))
+ template_path = File.join(templates_path, "tarantula_test.rb")
+ cp template_path, "test/tarantula/" , :verbose => true
+ blessed_path = File.join(templates_path, "blessed.rb")
+ cp blessed_path, "test/tarantula/", :verbose => true
end
end
View
9 lib/tarantula-rails3.rb
@@ -0,0 +1,9 @@
+module Relevance
+ module Tarantula
+ class Railtie < ::Rails::Railtie
+ rake_tasks do
+ load "relevance/tasks/tarantula_tasks.rake"
+ end
+ end
+ end
+end
View
BIN  pkg/tarantula-rails3-0.4.0.gem
Binary file not shown
View
144 tarantula-rails3.gemspec
@@ -0,0 +1,144 @@
+# Generated by jeweler
+# DO NOT EDIT THIS FILE DIRECTLY
+# Instead, edit Jeweler::Tasks in Rakefile, and run 'rake gemspec'
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+ s.name = %q{tarantula-rails3}
+ s.version = "0.4.0"
+
+ s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+ s.authors = [%q{Relevance, Inc.}, %q{Luca Invernizzi}]
+ s.date = %q{2011-08-19}
+ s.description = %q{A big hairy fuzzy spider that crawls your site, wreaking havoc}
+ s.email = [%q{opensource@thinkrelevance.com}, %q{invernizzi.l@gmail.com}]
+ s.extra_rdoc_files = [
+ "LICENSE",
+ "README.rdoc"
+ ]
+ s.files = [
+ "CHANGELOG",
+ "LICENSE",
+ "README.rdoc",
+ "Rakefile",
+ "VERSION.yml",
+ "examples/example_helper.rb",
+ "examples/relevance/core_extensions/ellipsize_example.rb",
+ "examples/relevance/core_extensions/file_example.rb",
+ "examples/relevance/core_extensions/response_example.rb",
+ "examples/relevance/core_extensions/test_case_example.rb",
+ "examples/relevance/tarantula/attack_handler_example.rb",
+ "examples/relevance/tarantula/basic_attack_example.rb",
+ "examples/relevance/tarantula/crawler_example.rb",
+ "examples/relevance/tarantula/form_example.rb",
+ "examples/relevance/tarantula/form_submission_example.rb",
+ "examples/relevance/tarantula/html_document_handler_example.rb",
+ "examples/relevance/tarantula/html_report_helper_example.rb",
+ "examples/relevance/tarantula/html_reporter_example.rb",
+ "examples/relevance/tarantula/invalid_html_handler_example.rb",
+ "examples/relevance/tarantula/io_reporter_example.rb",
+ "examples/relevance/tarantula/link_example.rb",
+ "examples/relevance/tarantula/log_grabber_example.rb",
+ "examples/relevance/tarantula/rails_integration_proxy_example.rb",
+ "examples/relevance/tarantula/result_example.rb",
+ "examples/relevance/tarantula/tidy_handler_example.rb",
+ "examples/relevance/tarantula/transform_example.rb",
+ "examples/relevance/tarantula_example.rb",
+ "laf/images/header_bg.jpg",
+ "laf/images/logo.png",
+ "laf/images/tagline.png",
+ "laf/javascripts/jquery-1.2.3.js",
+ "laf/javascripts/jquery-ui-tabs.js",
+ "laf/javascripts/jquery.tablesorter.js",
+ "laf/javascripts/tarantula.js",
+ "laf/stylesheets/tarantula.css",
+ "lib/relevance/core_extensions/ellipsize.rb",
+ "lib/relevance/core_extensions/file.rb",
+ "lib/relevance/core_extensions/metaclass.rb",
+ "lib/relevance/core_extensions/response.rb",
+ "lib/relevance/core_extensions/string_chars_fix.rb",
+ "lib/relevance/core_extensions/test_case.rb",
+ "lib/relevance/tarantula.rb",
+ "lib/relevance/tarantula/attack.rb",
+ "lib/relevance/tarantula/attack_handler.rb",
+ "lib/relevance/tarantula/basic_attack.rb",
+ "lib/relevance/tarantula/blessing.rb",
+ "lib/relevance/tarantula/crawler.rb",
+ "lib/relevance/tarantula/detail.html.erb",
+ "lib/relevance/tarantula/form.rb",
+ "lib/relevance/tarantula/form_submission.rb",
+ "lib/relevance/tarantula/html_document_handler.rb",
+ "lib/relevance/tarantula/html_report_helper.rb",
+ "lib/relevance/tarantula/html_reporter.rb",
+ "lib/relevance/tarantula/index.html.erb",
+ "lib/relevance/tarantula/invalid_html_handler.rb",
+ "lib/relevance/tarantula/io_reporter.rb",
+ "lib/relevance/tarantula/link.rb",
+ "lib/relevance/tarantula/log_grabber.rb",
+ "lib/relevance/tarantula/rails_integration_proxy.rb",
+ "lib/relevance/tarantula/recording.rb",
+ "lib/relevance/tarantula/response.rb",
+ "lib/relevance/tarantula/result.rb",
+ "lib/relevance/tarantula/test_report.html.erb",
+ "lib/relevance/tarantula/text_presence.rb",
+ "lib/relevance/tarantula/tidy_handler.rb",
+ "lib/relevance/tarantula/transform.rb",
+ "lib/relevance/tasks/tarantula_tasks.rake",
+ "lib/tarantula-rails3.rb",
+ "template/blessed.rb",
+ "template/tarantula_test.rb"
+ ]
+ s.homepage = %q{http://github.com/invernizzi/tarantula-rails3}
+ s.require_paths = [%q{lib}]
+ s.rubygems_version = %q{1.8.8}
+ s.summary = %q{A big hairy fuzzy spider that crawls your site, wreaking havoc}
+ s.test_files = [
+ "examples/example_helper.rb",
+ "examples/relevance/core_extensions/ellipsize_example.rb",
+ "examples/relevance/core_extensions/file_example.rb",
+ "examples/relevance/core_extensions/response_example.rb",
+ "examples/relevance/core_extensions/test_case_example.rb",
+ "examples/relevance/tarantula/attack_handler_example.rb",
+ "examples/relevance/tarantula/basic_attack_example.rb",
+ "examples/relevance/tarantula/crawler_example.rb",
+ "examples/relevance/tarantula/form_example.rb",
+ "examples/relevance/tarantula/form_submission_example.rb",
+ "examples/relevance/tarantula/html_document_handler_example.rb",
+ "examples/relevance/tarantula/html_report_helper_example.rb",
+ "examples/relevance/tarantula/html_reporter_example.rb",
+ "examples/relevance/tarantula/invalid_html_handler_example.rb",
+ "examples/relevance/tarantula/io_reporter_example.rb",
+ "examples/relevance/tarantula/link_example.rb",
+ "examples/relevance/tarantula/log_grabber_example.rb",
+ "examples/relevance/tarantula/rails_integration_proxy_example.rb",
+ "examples/relevance/tarantula/result_example.rb",
+ "examples/relevance/tarantula/tidy_handler_example.rb",
+ "examples/relevance/tarantula/transform_example.rb",
+ "examples/relevance/tarantula_example.rb"
+ ]
+
+ if s.respond_to? :specification_version then
+ s.specification_version = 3
+
+ if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+ s.add_runtime_dependency(%q<htmlentities>, [">= 4.2.0"])
+ s.add_runtime_dependency(%q<hpricot>, [">= 0.8.1"])
+ s.add_runtime_dependency(%q<tidy>, [">= 0"])
+ s.add_development_dependency(%q<micronaut>, [">= 0"])
+ s.add_development_dependency(%q<log_buddy>, [">= 0"])
+ else
+ s.add_dependency(%q<htmlentities>, [">= 4.2.0"])
+ s.add_dependency(%q<hpricot>, [">= 0.8.1"])
+ s.add_dependency(%q<tidy>, [">= 0"])
+ s.add_dependency(%q<micronaut>, [">= 0"])
+ s.add_dependency(%q<log_buddy>, [">= 0"])
+ end
+ else
+ s.add_dependency(%q<htmlentities>, [">= 4.2.0"])
+ s.add_dependency(%q<hpricot>, [">= 0.8.1"])
+ s.add_dependency(%q<tidy>, [">= 0"])
+ s.add_dependency(%q<micronaut>, [">= 0"])
+ s.add_dependency(%q<log_buddy>, [">= 0"])
+ end
+end
+
View
24 tarantula.gemspec
@@ -1,20 +1,20 @@
# -*- encoding: utf-8 -*-
Gem::Specification.new do |s|
- s.name = %q{tarantula}
+ s.name = %q{tarantula-rails3}
s.version = "0.3.3"
s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
- s.authors = ["Relevance, Inc."]
- s.date = %q{2009-09-24}
+ s.authors = ["Relevance, Inc.", "Luca Invernizzi"]
+ s.date = %q{2011-02-14}
s.description = %q{A big hairy fuzzy spider that crawls your site, wreaking havoc}
- s.email = %q{opensource@thinkrelevance.com}
+ s.email = [%q{opensource@thinkrelevance.com}, %q{invernizzi.l@gmail.com}]
s.extra_rdoc_files = [
"README.rdoc"
]
s.files = [
"CHANGELOG",
- "MIT-LICENSE",
+ "LICENSE",
"README.rdoc",
"Rakefile",
"VERSION.yml",
@@ -77,15 +77,18 @@ Gem::Specification.new do |s|
"lib/relevance/tarantula/test_report.html.erb",
"lib/relevance/tarantula/tidy_handler.rb",
"lib/relevance/tarantula/transform.rb",
- "tasks/tarantula_tasks.rake",
- "template/tarantula_test.rb"
+ "lib/relevance/tarantula/blessing.rb",
+ "lib/tarantula-rails3.rb",
+ "lib/relevance/tasks/tarantula_tasks.rake",
+ "template/tarantula_test.rb",
+ "template/blessed.rb"
]
- s.homepage = %q{http://github.com/relevance/tarantula}
+ s.homepage = %q{http://github.com/nashby/tarantula-rails3}
s.rdoc_options = ["--charset=UTF-8"]
s.require_paths = ["lib"]
s.rubyforge_project = %q{thinkrelevance}
- s.rubygems_version = %q{1.3.4}
- s.summary = %q{A big hairy fuzzy spider that crawls your site, wreaking havoc}
+ s.rubygems_version = %q{1.5.0}
+ s.summary = %q{A big hairy fuzzy spider that crawls your site, wreaking havoc. For Rails 3.}
s.test_files = [
"examples/example_helper.rb",
"examples/relevance/core_extensions/ellipsize_example.rb",
@@ -111,6 +114,7 @@ Gem::Specification.new do |s|
"examples/relevance/tarantula_example.rb"
]
+ s.add_dependency "tidy"
if s.respond_to? :specification_version then
current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
s.specification_version = 3
View
14 template/blessed.rb
@@ -0,0 +1,14 @@
+require "active_support/core_ext"
+module Blessings
+ mattr_accessor :blessings
+
+ # Append the errors you want to ignore to the following array.
+ # The format is:
+ # [ < URL | URL_REGEXP > , HASH ]
+ #
+ # See also these examples
+ self.blessings = [
+ ["/dashboard", "9ed37f41d6d123e805c3f00dbf6eb0a5"], # Suppress error X on http://example.com/dashboard
+ [/.*/, "baddeed"], # Suppress this error on every url
+ ]
+end
View
4 template/tarantula_test.rb
@@ -1,7 +1,9 @@
-require "#{File.dirname(__FILE__)}/../test_helper"
+require "test_helper"
require "relevance/tarantula"
class TarantulaTest < ActionController::IntegrationTest
+ include Relevance::CoreExtensions::TestCaseExtensions
+
# Load enough test data to ensure that there's a link to every page in your
# application. Doing so allows Tarantula to follow those links and crawl
# every page. For many applications, you can load a decent data set by
Something went wrong with that request. Please try again.