made output plugins more DRY, added mongodb logging features, updated…

… mysql_connect plugin, updated putty-log plugin

Added MongoDB logging. Use with --log-mongo-database, --log-mongo-host and --log-mongo-collection.

CHANGELOG

@@ -40,7 +40,7 @@ Version 0.4.6 Released ? 2010
 * Updated mailto plugin
 * Updated Mambo and Joomla plugins with suggestions from Aung Khant
 * Bug fix: Updated WordPress plugin
-* Added mongo support - MongoOutput and --log-mongo
+* Added MongoDB logging. Use with --log-mongo-database, --log-mongo-host and --log-mongo-collection. Only database has no default.
 * Added Charset plugin, required for conversion to UTF-8 which is required for MongoDB. Optionally requires rchardet gem
 * Added vulnerability matching support, this is still in the experimental phase
 * Added vulerability matching code to awstats.
@@ -60,6 +60,7 @@ Version 0.4.6 Released ? 2010
 * Updated Mailto plugin, renamed to Email
 * Bug fix: Now redirects for HTTP statuses 300 through 399. Previously redirected for 301,302 and 307.
 
+
 Version 0.4.5 Released August 17th 2010
 * Added 5 plugins from Tonmoy Saikia. They are: Commonspot, TextPattern, Mediawiki, DUclassified and Mailman
 * Added 119 plugins from Brendan Coles. They are: Alcatel-Lucent-Omniswitch, Allinta-CMS, anyInventory, Arab-Portal, AVTech-Video-Web-Server, Barracuda-Spam-Firewall, Basilic, Biromsoft-WebCam, BlueNet-Video-Server, BM-Classifieds, Brother-Printer, BusinessSpace, BXR, Campsite, Canon-Network-Camera, Cisco-VPN-3000-Concentrator, CMSQLite, ColdFusion, coWiki, cpCommerce, CruxCMS, CruxPA, Dell-Printer, D-Link-Network-Camera, DMXReady, DT-Centrepiece, EazyCMS, eLitius, EMO-Realty-Manager, Empire-CMS, envezion~media, eSyndiCat, Evo-Cam, FestOS, Flax-Article-Manager, FluentNET, Forest-Blog, GuppY, HP-LaserJet-Printer, i-Catcher-Console, iDVR, Intellinet-IP-Camera, Interspire-Shopping-Cart, IPCop-Firewall, IQeye-Netcam, iRealty, iScripts-CyberMatch, iScripts-EasySnaps, iScripts-MultiCart, iScripts-ReserveLogic, iScripts-SocialWare, JAMM-CMS, Jamroom, Linksys-NAS, Linksys-Network-Camera, Linksys-Wireless-G-Camera, LocazoList-Classifieds, Lucky-Tech-iGuard, Mobotix-Network-Camera, MyioSoft-Ajax-Portal, My-PHP-Indexer, My-WebCamXP-Server, NetBotz-Network-Monitoring-Device, Netious-CMS, Netsnap-Web-Camera, Nukedit, Open-Blog, ORCA-Platform, ORITE-301-Camera, PageUp-People, Panasonic-Network-Camera, Parked-Domain, PHPDirector, PHPEasyData, phPhotoAlbum, Pixel-Ads-Script, Pixie, Pligg-CMS, PortalApp, Pressflow, RunCMS, sabros.us, samPHPweb, SHOUTcast-Administrator, SimpNews, SkaLinks, SmodCMS, Snap-Appliance-Server, Softbiz-Freelancers-Script, Softbiz-Online-Auctions-Script, Softbiz-Online-Classifieds, Sony-Network-Camera, Sony-Video-Network-Station, Stardot-Express, StarDot-NetCam, Star-Network, Subdreamer-CMS, Subrion-CMS, SyndeoCMS, syntaxCMS, TaskFreak, Team-Board, The-PHP-Real-Estate-Script, TomatoCMS, Toshiba-Network-Camera, Veo-Observer, VisionGS-Webcam, WebDVR, WebEye-Network-Camera, WebPress, WhiteBoard, Winamp-Web-Interface, Windows-Internet-Printing, Xerox-Printers, xGB, XHP-CMS, Zeus-Cart, Zoph, Zyxel-Vantage-Service-Gateway

TODO

@@ -21,8 +21,16 @@ Ruby1.9 compatability
 [x]	--plugins +/tmp/moo.rb
 [x]	--plugins foobar (only select foobar)
 
+[x] currently regex's like :modules only return the first instance - should be all.
+[x] add more mongo logging options  --mongo-host  default: 0.0.0.0   --mongo-database  default: test  --mongo-collection
+
+
+
+update lib/output.rb verbose logging. some plugin names are longer than 30
+optino 1)  truncate output name
+option 2) wrap plugin name to two lines
+option 3) break columns with that 1 plugin name
 
-currently regex's like :modules only return the first instance - should be all.
 
 
 Change usage:
@@ -39,6 +47,11 @@ Change usage:
 move this block to a pre output stage, be more DRY:
   string = plugin_results.map {|x| x[:string] unless x[:string].class==Regexp }.compact.sort.uniq.join(",")
 
+diff output plugins require - joined by , or not
+
+
+
+deprecate :account in favour of :accounts
 
 set num of max redirects, instead of hard coded value of 10
 

lib/output.rb

@@ -28,6 +28,18 @@ def initialize(f=STDOUT)
 	def close
 		@f.close unless @f.class == IO
 	end
+
+	# perform sort, uniq and join on each plugin result
+	def suj(plugin_results)
+		suj={} 
+		[:certainty, :version, :string, :account, :accounts, :model, :firmware, :modules, :filepath].map do  |thissymbol|
+			t=plugin_results.map {|x| x[thissymbol] unless x[thissymbol].class==Regexp }.compact.sort.uniq.join(",")
+			suj[thissymbol] = t
+		end
+		suj[:certainty] = plugin_results.map {|x| x[:certainty] }.compact.sort.last.to_i # this is different, it's a number
+		suj
+	end
+
 end
 
 class OutputFull < Output
@@ -43,7 +55,7 @@ class OutputVerbose < Output
 	def out(target, status, results)
 		results.each do |plugin_name,plugin_results|
 			unless plugin_results.empty?
-				@f.print plugin_name + " " * (30- plugin_name.size )+   " => "
+				@f.print plugin_name + " " * (40- plugin_name.size )+   " => "
 				matches = plugin_results.map do |pr|					
 					if pr[:name]
 						name_of_match = pr[:name]
@@ -90,16 +102,10 @@ def out(target, status, results)
 
 		results.each do |plugin_name,plugin_results|
 			unless plugin_results.empty?
-				# important info in brief mode is version, type and ?
-				# what's the highest probability for the match?
-				certainty = plugin_results.map {|x| x[:certainty] unless x[:certainty].class==Regexp }.compact.sort.uniq.last					
-				version = plugin_results.map {|x| x[:version] unless x[:version].class==Regexp }.compact.sort.uniq.join(",")
-				string = plugin_results.map {|x| x[:string] unless x[:string].class==Regexp }.compact.sort.uniq.join(",")
-				accounts = plugin_results.map {|x| [x[:account],x[:accounts] ] }.flatten.compact.sort.uniq.join(",")
-				model = plugin_results.map {|x| x[:model] unless x[:model].class==Regexp }.compact.sort.uniq.join(",")
-				firmware = plugin_results.map {|x| x[:firmware] unless x[:firmware].class==Regexp }.compact.sort.uniq.join(",")
-				modules = plugin_results.map {|x| x[:modules] unless x[:modules].class==Regexp }.compact.sort.uniq.join(",")
-				filepath = plugin_results.map {|x| x[:filepath] unless x[:filepath].class==Regexp }.compact.sort.uniq.join(",")
+				suj = suj(plugin_results)
+
+				certainty, version, string, accounts,model,firmware,modules,filepath = suj[:certainty],suj[:version],suj[:string], suj[:accounts],suj[:model],suj[:firmware],suj[:modules],suj[:filepath]
+
 
 				# be more DRY		
 				# if plugins have categories or tags this would be better, eg. all hash plugins are grey
@@ -303,9 +309,13 @@ def out(target, status, results)
 # basically the same as OutputJSON
 class OutputMongo < Output
 
-	def initialize(collection)
-	# should make databse and collection comma or fullstop delimited, eg. test,scan
-		@db = Mongo::Connection.new("0.0.0.0").db("test") # resolve-replace means we can't connect to localhost by name
+	def initialize(s)		
+		host=s[:host] || "0.0.0.0"
+		database=s[:database] || raise("Missing MongoDB database name")
+		collection=s[:collection] || "whatweb"
+
+		# should make databse and collection comma or fullstop delimited, eg. test,scan
+		@db = Mongo::Connection.new(host).db(database) # resolve-replace means we can't connect to localhost by name and must use 0.0.0.0
 		@coll=@db.collection(collection)
 		@charset=nil
 	end
@@ -385,11 +395,15 @@ def out(target, status, results)
 		end
 
 		@charset=results.map {|n,r| r[0][:string] if n=="Charset" }.compact.first
+
 		unless @charset.nil? or @charset == "Failed"
 			utf8_elements!(foo) # convert foo to utf-8
 			flatten_elements!(foo)
 			@coll.insert(foo)
+		else
+			error("#{target}: Failed to detect Character set and log to MongoDB")
 		end
+
 	end
 end
 

plugins/PuTTY-log.rb

@@ -64,7 +64,7 @@ def passive
 		if @body =~ /^Event Log: Writing new session log \(SSH packets mode\) to file: /
 			if @body =~ /^Event Log: Looking up host "([^\"]+)"/
 				account=@body.scan(/^Event Log: Looking up host "([^\"]+)"/)
-				m << {:account=>account}
+				m << {:accounts=>account}
 			end
 			if @body =~ /^Event Log: Server version:[\s]+([^\s]+)/
 				version=@body.scan(/^Event Log: Server version:[\s]+([^\s]+)/)

plugins/mysql_connect.rb

@@ -4,9 +4,12 @@
 # web site for more information on licensing and terms of use.
 # http://www.morningstarsecurity.com/research/whatweb
 ##
+
+# Version 0.2 Haiku edits by Andrew Horton
+
 Plugin.define "mysql_connect" do
 author "Brendan Coles <bcoles@gmail.com>" # 2011-01-19
-version "0.1"
+version "0.2"
 description "This plugin detects instances of the mysql_connect() function in PHP source code and retrieves the mysql server hostname, username and password if it's in plain-text. Alternatively, if the connection details are stored in variables it will return the variable names with possible values for those variables returned in :string=>"
 
 # 338 results for mysql_connect ext:inc  @ 2011-01-19
@@ -33,43 +36,52 @@
 
 # Passive #
 # Detect instances of the mysql_connect function and extract details
-# The code is pretty ugly but it gets the job done. Improvements welcomed.
 def passive
 	m=[]
+	hup={}; hup[:host] = []; hup[:user] = []; hup[:pass] = []
+
+	stuff={
+		:host=>/mysql_connect\([\s]*([^\r^\n^\)]*),[\s]*[^\r^\n^\)]*,[\s]*[^\r^\n^\)]*\)[^\r^\n^;]*;/,
+		:user=>/mysql_connect\([\s]*[^\r^\n^\)]*,[\s]*([^\r^\n^\)]*),[\s]*[^\r^\n^\)]*\)[^\r^\n^;]*;/,
+		:pass=>/mysql_connect\([\s]*[^\r^\n^\)]*,[\s]*[^\r^\n^\)]*,[\s]*([^\r^\n^\)]*)\)[^\r^\n^;]*;/
+	}
 
 	# Detect mysql_connect() function
 	if @body =~ /mysql_connect\([^\r^\n^\)]*,[\s]*[^\r^\n^\)]*,[\s]*[^\r^\n^\)]*\)[^\r^\n^;]*;/
 
-		# Extract host(s)
-		hosts=@body.scan(/mysql_connect\([\s]*([^\r^\n^\)]*),[\s]*[^\r^\n^\)]*,[\s]*[^\r^\n^\)]*\)[^\r^\n^;]*;/)
-		hosts.each do |line|
-			m << { :model=>line }
-			if line.to_s =~ /^[\s]*\$[\w_]+/
-				r=Regexp.new("[\s]*"+Regexp.escape(line.to_s)+"[\s]*=[\s]*([^\r^\n]*);")
-				m << { :string=>@body.scan(r) } if @body =~ r
+		stuff.each do |symbol,regex|
+			@body.scan(regex).each do |line|
+				hup[symbol] << line
+				if line.to_s =~ /^[\s]*\$[\w_]+/
+					r=Regexp.new("[\s]*"+Regexp.escape(line.to_s)+"[\s]*=[\s]*([^\r^\n]*);")
+					if @body =~ r
+						found=@body.scan(r)
+						if found.size > 1
+							hup[symbol] << found.join("+") # u want to change this?
+						else
+							hup[symbol] << found.first
+						end
+					end
+				end
 			end
 		end
+
+		unless hup[:user].empty?
+			# haiku
 
-		# Extract username(s)
-		usernames=@body.scan(/mysql_connect\([\s]*[^\r^\n^\)]*,[\s]*([^\r^\n^\)]*),[\s]*[^\r^\n^\)]*\)[^\r^\n^;]*;/)
-		usernames.each do |line|
-			m << { :firmware=>line }
-			if line.to_s =~ /^[\s]*\$[\w_]+/
-				r=Regexp.new("[\s]*"+Regexp.escape(line.to_s)+"[\s]*=[\s]*([^\r^\n]*);")
-				m << { :string=>@body.scan(r) } if @body =~ r
-			end
-		end
+			# plugin changed, edit
+			# rewritten sourcecode patterns
+			# succint, better now
 
-		# Extract password(s)
-		passwords=@body.scan(/mysql_connect\([\s]*[^\r^\n^\)]*,[\s]*[^\r^\n^\)]*,[\s]*([^\r^\n^\)]*)\)[^\r^\n^;]*;/)
-		passwords.each do |line|
-			m << { :filepath=>line }
-			if line.to_s =~ /^[\s]*\$[\w_]+/
-				r=Regexp.new("[\s]*"+Regexp.escape(line.to_s)+"[\s]*=[\s]*([^\r^\n]*);")
-				m << { :string=>@body.scan(r) } if @body =~ r
-			end
-		end
+# hup ends up like this:
+# {:pass=>[["\"\""], ["'680dgg%y'"]], :host=>[["\"localhost\""], ["'localhost'"]], :user=>[["\"root\""], ["'root'"]]}
 
+			hup.values.each {|x| x.each {|y| y.first.gsub!(/^['"]|['"]$/,'') }} # remove the ' and "
+
+			ret=(0..hup[:host].size-1).map {|x|
+				[hup[:host][x],hup[:user][x],hup[:pass][x]].join(",") }.map {|x| '(' + x + ')' }.join(",")
+			m << { :string=>ret }
+		end
 	end
 
 	m

whatweb

@@ -592,8 +592,7 @@ def open_target(target)
 			http.verify_mode = OpenSSL::SSL::VERIFY_NONE		
 		end
 
-		req=Net::HTTP::Get.new(path + (query.nil? ? "" : "?" + query ) ,{"User-Agent"=>$USER_AGENT})
-
+		req=Net::HTTP::Get.new(path + (query.nil? ? "" : "?" + query ) ,{"User-Agent"=>$USER_AGENT})	
 		res=http.request(req)
 
 		headers={}; res.each_header {|x,y| headers[x]=y }
@@ -608,6 +607,9 @@ def open_target(target)
 	rescue TimeoutError => err
 		error(target + " ERROR: Timed out #{err}")
 		return [0, nil, nil, nil,nil]
+	rescue Errno::ETIMEDOUT	=>err # for ruby 1.8.7 patch level 249
+		error(target + " ERROR: Timed out (ETIMEDOUT) #{err}")
+		return [0, nil, nil, nil,nil]
 	rescue EOFError => err
 		error(target + " ERROR: EOF error #{err}")
 		return [0, nil, nil, nil,nil]
@@ -748,7 +750,9 @@ puts "
 --log-xml=FILE\t\tLog XML format
 --log-json=FILE\t\tLog JSON format
 --log-json-verbose=FILE\tLog JSON Verbose format
---log-mongo=Collection\tLog to a Mongo DB on localhost
+--log-mongo-database\tName of the MongoDB database
+--log-mongo-collection\tName of the MongoDB collection. Default: whatweb
+--log-mongo-host\tMongoDB hostname or IP address. Default: 0.0.0.0
 --log-errors=FILE\tLog errors
 --user-agent, -U\tIdentify as user-agent instead of WhatWeb/#{$VERSION}.
 --max-threads, -t\tNumber of simultaneous threads. Default is #{$MAX_THREADS}.
@@ -776,14 +780,17 @@ puts "
 --verbose, -v\t\tIncrease verbosity, use twice for debugging.
 --debug\t\t\tRaise errors in plugins
 --version\t\tDisplay version information. (WhatWeb #{$VERSION})\n\n"
-end
-
-
-
-
-
 
+	suggestions=""
+	suggestions << "To boost performance during long scans install the em-resolve-replace gem.\n" unless Gem.available?('em-resolv-replace')
+	suggestions << "To enable JSON logging install the json gem.\n" unless Gem.available?('json')
+	suggestions << "To enable MongoDB logging install the mongo gem.\n" unless Gem.available?('mongo')
+	suggestions << "To enable character set detection and MongoDB logging install the rchardet gem.\n" unless Gem.available?('rchardet')
 
+	unless suggestions.empty?
+		print "\nOptional dependencies:\n" + suggestions + "\n"
+	end
+end
 
 
 if ARGV.size==0 # faster usage info
@@ -794,6 +801,10 @@ end
 plugin_selection=nil
 input_file=nil
 output_list = []
+mongo={}
+mongo[:use_mongo_log]=false
+
+
 
 opts = GetoptLong.new(
       [ '--help', '-h', GetoptLong::NO_ARGUMENT ],
@@ -808,7 +819,9 @@ opts = GetoptLong.new(
       [ '--log-xml', GetoptLong::REQUIRED_ARGUMENT ],
       [ '--log-json', GetoptLong::REQUIRED_ARGUMENT ],
       [ '--log-json-verbose', GetoptLong::REQUIRED_ARGUMENT ],
-      [ '--log-mongo', GetoptLong::REQUIRED_ARGUMENT ],
+      [ '--log-mongo-collection', GetoptLong::REQUIRED_ARGUMENT ],
+      [ '--log-mongo-host', GetoptLong::REQUIRED_ARGUMENT ],
+      [ '--log-mongo-database', GetoptLong::REQUIRED_ARGUMENT ],
       [ '--log-errors', GetoptLong::REQUIRED_ARGUMENT ],
       [ '-i','--input-file', GetoptLong::REQUIRED_ARGUMENT ],
       [ '-U','--user-agent', GetoptLong::REQUIRED_ARGUMENT ],
@@ -879,9 +892,26 @@ begin
 				else
 					abort("Sorry. The JSON gem is required for JSONVerbose output")
 				end
-			when '--log-mongo'
+			when '--log-mongo-collection'
 				if defined?(Mongo) and defined?(CharDet)
-			 		output_list << OutputMongo.new(arg)
+					mongo[:collection]=arg
+					mongo[:use_mongo_log]=true
+				else
+					abort("Sorry. The mongo and rchardet gems are required for Mongo output")
+				end
+
+			when '--log-mongo-host'
+				if defined?(Mongo) and defined?(CharDet)
+			 		mongo[:host]=arg
+					mongo[:use_mongo_log]=true
+				else
+					abort("Sorry. The mongo and rchardet gems are required for Mongo output")
+				end
+
+			when '--log-mongo-database'
+				if defined?(Mongo) and defined?(CharDet)
+			 		mongo[:database]=arg
+					mongo[:use_mongo_log]=true
 				else
 					abort("Sorry. The mongo and rchardet gems are required for Mongo output")
 				end
@@ -948,11 +978,6 @@ rescue GetoptLong::Error => err
 	exit
 end
 
-### OUTPUT
-output_list << OutputBrief.new unless $QUIET # by default output brief
-output_list << OutputFull.new() if $verbose > 1 # full output if -vv
-output_list <<  OutputVerbose.new() if $verbose > 0 # full output if -v
-
 
 ### PLUGINS
 $plugins_to_use = load_plugins(plugin_selection) # load all the plugins
@@ -965,6 +990,21 @@ end
 precompile_regular_expressions # optimise plugins
 
 
+### OUTPUT
+output_list << OutputBrief.new unless $QUIET # by default output brief
+output_list << OutputFull.new() if $verbose > 1 # full output if -vv
+output_list <<  OutputVerbose.new() if $verbose > 0 # full output if -v
+
+if mongo[:use_mongo_log]
+	if $plugins_to_use.map { |a,b| a }.include?("Charset")
+		output_list << OutputMongo.new(mongo) 
+	else
+		error("MongoDB logging requires the Charset plugin to be activated. It is not included by default for speed considerations.")
+		exit
+	end
+end
+
+
 ### TARGETS
 # clean up urls, add example urls if needed
 $targets=make_target_list(ARGV, input_file, $plugins_to_use)