Permalink
Browse files

included the MU/Xtractr libraries, for an easier deployment

readme fixed
table traffic_dumps fixes
dparser fixed (added exceptions)
  • Loading branch information...
1 parent b33aa68 commit 245119c2cf7003d3302429d1d65639619fdb9e72 @m4rco- committed Jun 4, 2013
View
@@ -13,7 +13,7 @@ Dorothy2 is a continuation of my Bachelor degree's final project ([Dorothy: insi
The main framework's structure remained almost the same, and it has been fully detailed in my degree's final project or in this short [paper](http://www.honeynet.it/wp-content/uploads/Dorothy/EC2ND-Dorothy.pdf). More information about the whole project can be found on the Italian Honeyproject [website](http://www.honeynet.it).
-The framework is manly composed by four big elements that can be even executed separately:
+The framework is mainly composed by four big elements that can be even executed separately:
* The Dorothy analysis engine (included in this gem)
@@ -33,7 +33,7 @@ The framework is manly composed by four big elements that can be even executed s
The first three modules are (or will be soon) publicly released under GPL 2/3 license as tribute to the the [Honeynet Project Alliance](http://www.honeynet.org).
All the information generated by the framework - i.e. binary info, timestamps, dissected network analysis - are stored into a postgres DB (Dorothive) in order to be used for further analysis.
-A no-SQL database (CouchDB) is also used to mass strore all the traffic dumps thanks to the [pcapr/xtractr](https://code.google.com/p/pcapr/wiki/Xtractr) technology.
+A no-SQL database (CouchDB) is also used to mass store all the traffic dumps thanks to the [pcapr/xtractr](https://code.google.com/p/pcapr/wiki/Xtractr) technology.
I started to code this project in late 2009 while learning Ruby at the same time. Since then, I´ve been changing/improving it as long as my Ruby coding skills were improving. Because of that, you may find some parts of code not-really-tidy :)
@@ -164,7 +164,7 @@ or
2. Configure a dedicated postgres user for Dorothy (or use the default postgres user instead, up to you :)
> Note:
-> If you want to use Postgres "as is", and then configure Dorothy to use "postgres" degault the user, configure a password for this user at least (by default it comes with no password)
+> If you want to use Postgres "as is", and then configure Dorothy to use "postgres" default the user, configure a password for this user at least (by default it comes with no password)
3. Install the following packages
View
@@ -6,9 +6,9 @@
require 'rubygems'
require 'trollop'
-require 'dorothy2' #comment for testing/developmnet
+#require 'dorothy2' #comment for testing/developmnet
-#load '../lib/dorothy2.rb' #uncomment for testing/developmnet
+load '../lib/dorothy2.rb' #uncomment for testing/developmnet
include Dorothy
View
@@ -7,9 +7,9 @@
require 'rubygems'
require 'trollop'
require 'dorothy2'
-require 'doroParser'
+#require 'doroParser'
-#load '../lib/doroParser.rb'
+load '../lib/doroParser.rb'
include Dorothy
include DoroParser
View
@@ -298,7 +298,7 @@ COMMENT ON CONSTRAINT size_notneg ON samples IS 'Sample size must not be negativ
CREATE TABLE traffic_dumps (
hash character(64) NOT NULL,
size integer NOT NULL,
- pcapr_id character(64),
+ pcapr_id character(32),
"binary" character varying,
parsed boolean
);
View
@@ -17,7 +17,6 @@
require 'rubygems'
-require 'mu/xtractr'
require 'md5'
require 'rbvmomi'
require 'rest_client'
@@ -32,15 +31,17 @@
require 'iconv'
require 'tmail'
require 'ipaddr'
+require 'net/http'
+require 'json'
require File.dirname(__FILE__) + '/dorothy2/environment'
+require File.dirname(__FILE__) + '/mu/xtractr'
require File.dirname(__FILE__) + '/dorothy2/DEM'
require File.dirname(__FILE__) + '/dorothy2/do-utils'
require File.dirname(__FILE__) + '/dorothy2/do-logger'
require File.dirname(__FILE__) + '/dorothy2/deep_symbolize'
-
module DoroParser
#Host roles
@@ -85,12 +86,34 @@ def analyze_bintraffic(pcaps)
begin
- xtractr = Doroxtractr.create "http://#{DoroSettings.pcapr[:host]}:#{DoroSettings.pcapr[:port]}/pcaps/1/pcap/#{dump['pcapr_id'].gsub(/\s+/, "")}"
+
+ #check if the pcap has been correctly indexed by pcapr
+ xtractr = Doroxtractr.create "http://#{DoroSettings.pcapr[:host]}:#{DoroSettings.pcapr[:port]}/pcaps/1/pcap/#{dump['pcapr_id'].rstrip}"
rescue => e
- LOGGER_PARSER.fatal "PARSER", "Can't create a XTRACTR instance, try with nextone"
+ LOGGER_PARSER.fatal "PARSER", "Can't connect to the PCAPR server."
LOGGER_PARSER.debug "PARSER", "#{$!}"
- LOGGER_PARSER.debug "PARSER", e
+ LOGGER_PARSER.debug "PARSER", e.backtrace if VERBOSE
+ return false
+ end
+
+ #it may happen that Pcapr has created an instance, but it is still indexing the pcap.
+ #The following section is to avoid a crash while quering such (still-empty instance)
+ #In addition, an added check is inserted, to see if the pcapr instance really match the pcap filename
+ begin
+ pcapr_query = URI.parse "http://#{DoroSettings.pcapr[:host]}:#{DoroSettings.pcapr[:port]}/pcaps/1/about/#{dump['pcapr_id'].rstrip}"
+ pcapr_response = Net::HTTP.get_response(pcapr_query)
+ pcapname = File.basename(JSON.parse(pcapr_response.body)["filename"], ".pcap")
+
+ t ||= $1 if pcapname =~ /[0-9]*\-(.*)$/
+ raise NameError.new if t != dump['sample'].rstrip
+
+ rescue NameError
+ LOGGER_PARSER.error "PARSER", "The pcapr filename mismatchs the one present in Dorothive!. Skipping."
+ next
+
+ rescue
+ LOGGER_PARSER.error "PARSER", "Can't find the PCAP into Pcapr, maybe it has not been indexed yet. Skipping."
next
end
@@ -103,18 +126,14 @@ def analyze_bintraffic(pcaps)
flowdeep = xtractr.flows("flow.id:#{flow.id}")
-
-
#Skipping if NETBIOS spreading activity:
if flow.dport == 135 or flow.dport == 445
LOGGER_PARSER.info "PARSER", "Netbios connections, skipping flow" unless NONETBIOS
next
end
-
title = flow.title[0..200].gsub(/'/,"") #xtool bug ->')
-
#insert hosts (geo) info into db
#TODO: check if is a localaddress
localip = xtractr.flows.first.src.address
@@ -258,19 +277,13 @@ def analyze_bintraffic(pcaps)
end
-
-
-
-
end
#case MAIL
when "SMTP" then
LOGGER_PARSER.info "SMTP", "FOUND an SMTP request..".white
#insert mail
#by from to subject data id time connection
-
-
streamdata.each do |m|
mailfrom = 'null'
mailto = 'null'
@@ -303,8 +316,6 @@ def analyze_bintraffic(pcaps)
@insertdb.insert("emails", mailvalues )
end
-
-
#case FTP
when "FTP" then
LOGGER_PARSER.info "FTP", "FOUND an FTP request".white
@@ -324,9 +335,7 @@ def analyze_bintraffic(pcaps)
end
end
-
else
-
LOGGER_PARSER.info "PARSER", "Unknown traffic, try see if it is IRC traffic"
if Parser.guess(streamdata.inspect).class.inspect =~ /IRC/
@@ -358,14 +367,12 @@ def analyze_bintraffic(pcaps)
end
end
-
@p.each do |d|
begin
dns = DoroDNS.new(d)
-
dnsvalues = ["default", dns.name, dns.cls_i.inspect, dns.qry?, dns.ttl, flowid, dns.address.to_s, dns.data, dns.type_i.inspect]
LOGGER_PARSER.debug "DB", " Inserting DNS data from #{flow.dst.address.to_s}".blue if VERBOSE
@@ -404,8 +411,8 @@ def analyze_bintraffic(pcaps)
rescue => e
- LOGGER_PARSER.error "PARSER", "Error while analyzing flow #{flow.id}"
- LOGGER_PARSER.debug "PARSER", "#{e.inspect} BACKTRACE: #{e.backtrace}"
+ LOGGER_PARSER.error "PARSER", "Error while analyzing flow #{flow.id}: #{e.inspect}"
+ LOGGER_PARSER.debug "PARSER", "#{e.backtrace}" if VERBOSE
LOGGER_PARSER.info "PARSER", "Flow #{flow.id} will be skipped"
next
end
View
@@ -5,7 +5,6 @@
##for irb debug:
##from $home, irb and :
##load 'lib/dorothy2.rb'; include Dorothy; LOGGER = DoroLogger.new(STDOUT, "weekly"); DoroSettings.load!('etc/dorothy.yml')
-#$LOAD_PATH.unshift '/opt/local/lib/ruby/gems/1.8/gems/ruby-filemagic-0.4.2/lib'
require 'net/ssh'
require 'net/scp'
@@ -152,8 +151,8 @@ def analyze(bin, guestvm)
vsm.copy_file("#{bin.md5}#{bin.extension}",filecontent)
#Start Sniffer
- dumpname = bin.md5
- pid = @nam.start_sniffer(guestvm[2],DoroSettings.nam[:interface], dumpname, DoroSettings.nam[:pcaphome]) #dumpname = vmfile.pcap
+ dumpname = anal_id.to_s + "-" + bin.md5
+ pid = @nam.start_sniffer(guestvm[2],DoroSettings.nam[:interface], dumpname, DoroSettings.nam[:pcaphome])
LOGGER.info "NAM","VM#{guestvm[0]} ".yellow + "Start sniffing module"
LOGGER.debug "NAM","VM#{guestvm[0]} ".yellow + "Tcpdump instance #{pid} started" if VERBOSE
@@ -216,8 +215,7 @@ def analyze(bin, guestvm)
#Downloading PCAP
LOGGER.info "NAM", "VM#{guestvm[0]} ".yellow + "Downloading #{dumpname}.pcap to #{bin.dir_pcap}"
- #t = DoroSettings.nam[:pcaphome] + "/" + dumpname + ".pcap"
- Ssh.download(DoroSettings.nam[:host], DoroSettings.nam[:user],DoroSettings.nam[:pass], DoroSettings.nam[:pcaphome] + "/" + dumpname + ".pcap", bin.dir_pcap)
+ Ssh.download(DoroSettings.nam[:host], DoroSettings.nam[:user],DoroSettings.nam[:pass], DoroSettings.nam[:pcaphome] + "/#{dumpname}.pcap", bin.dir_pcap)
#Downloading Screenshots from esx
LOGGER.info "NAM", "VM#{guestvm[0]} ".yellow + "Downloading Screenshots"
@@ -231,11 +229,10 @@ def analyze(bin, guestvm)
#UPDATE DOROTHIBE DB#
#####################
- pcapfile = bin.dir_pcap + dumpname + ".pcap"
- dump = Loadmalw.new(pcapfile)
+ dump = Loadmalw.new(bin.dir_pcap + dumpname + ".pcap")
#pcaprpath = bin.md5 + "/pcap/" + dump.filename
- pcaprid = Loadmalw.calc_pcaprid(dump.filename, dump.size)
+ pcaprid = Loadmalw.calc_pcaprid(dump.filename, dump.size).rstrip
LOGGER.debug "NAM", "VM#{guestvm[0]} ".yellow + "Pcaprid: " + pcaprid if VERBOSE
View
@@ -1,3 +1,3 @@
module Dorothy2
- VERSION = "0.0.2"
+ VERSION = "0.0.3"
end
Oops, something went wrong.

0 comments on commit 245119c

Please sign in to comment.