Every new device added to a network brings with it a new set of vulnerabilities that pose a threat to the entire system.To keep a track of what is added to the network, figuring out the threats introduced and implementing corresponding defenses is tedious as well as an error prone process.Given that vulnerabilities of network devices are known, we plan to use this information to identify dangerous packets and report them. This entire project involves building an algorithm that would first identify threats/attack modules for any device OS that is being added to the network, correlate these threats/attack modules to their respective CVEs and finally either use the Snort community rule files to look for pre-exisiting rules, or craft the Snort rule based on the payload, which will alert the network to malicious traffic. This ensures that the network has dynamic defense setup and is updated against known threats.
Design We intend to show how we can alert our network against known attacks on devices, without manual intervention. For this purpose, our design includes victim machines, which represent various new and un-patched devices that are added into the network. Snort IDS is used to detect and report an ongoing attack. Our Algorithm looks to leverage the Community Rules of Snort, by first looking at automatically determining the Vulnerabilities of the OS of the device on Metasploit and using the CVEs of those known vulnerabilities to filter and add corresponding Snort Rule to flag any potential malicious traffic. In order to increase the percentage of Modules having a direct Snort Rule correlation, we have also considered reading all previous versions of the Community Rulesets if the rule is absent in the latest version of the community ruleset. We have also looked to extend the algorithm to solve injection based overflow attacks due to a common signature being found across those attacks that we exploited.
Infrastructure and Testbed Setup Our testbed has two major types of nodes - attacker and the victims. Each of these VMs are on TopoMojo environment as shown in Figure 1.
Attacker: The first node in our system is the Attacker VM. This is a Kali Linux with Metasploit installed. We use this to scan the victim for exposed vulnerabilities and carry out the actual attack.
Victim: The second type of node in our system are the victim nodes.For our testing we have Windows Server 2008 R2, Windows 2012 R2 and Windows 2016 R2. We started out with this as, these machines have many vulnerabilities.
Snort: Last component of our setup is Snort. Snort is an open source network intrusion detection and prevention tool. This is placed between the attacker and the victim. We use Snort as a packet sniffer that monitors network traffic coming to the victim, in real time and scrutinizes packets to detect various threat signatures. Our algorithm to generate the Snort Rules for different vulnerabilities based on OS will be run here.
Due to testbed limitations, we don’t have an independent Snort VM between Kali and Windows. It is included as part of the Linux machine, which is the Attacker VM. Snort can be configured to block the attack. However, for the purpose of our project, we have used Snort to passively detect and report an attack.
Implementation Figure 2 explains the algorithm used to report an attack on our dynamic network. As and when a new machine is added to the network, the OS version is detected and taken as input in the snort algorithm. Based on the OS, we find the list vulnerabilities applicable to that version. Eventually we apply the set of corresponding snort rules, to detect and report known attacks that can be carried out on this OS version. Our assumption is that the attacker would use tools like Metasploit to find vulnerabilities in the device and these are the vulnerabilities that we need to protect our setup against.
Figure above explains the algorithm used to report an attack on our dynamic network. As a new machine is added, the OS version is detected and taken as input in the snort algorithm. Based on the OS, we find its vulnerabilities and a set of corresponding snort rules are applied, to detect and report known attacks that can be carried out on this OS.
Input: When a new system gets added in the our setup, we periodically poll the network to detect the type and OS version of the newly added node. We take this as an input file in our algorithm. Finding CVEs: From the OS version obtained above, we query Metasploit database to get all or top few vulnerabilities(CVEs) available for that OS version using Metasploit search and info commands in our code. Finding Snort Ruleset: We query the Snort community ruleset for the CVE. If found, we append the snort rules in a dictionary, corresponding to it’s vulnerable OS.
Finding Snort rules in Git Repositories: Snort community rules is not exhaustive. Preliminary results using Community Rulesets gave an accuracy of 15% in finding a direct mapping of a CVE to it’s snort rules. Using unofficial Github Repository which contains old community ruleset increased CVE coverage immensely and also allowed to extend our code’s knowledge base easily. Our assumption here is that the Git repository itself is reliable even though it is not official. OS to Snort rule mapping: Finally, map the OS version to the corresponding snort rules. Store in file: After the vulnerability dictionary is created, we write these rules in local_test.rules files. Rules in this file are used by the IDS to detect the attack.
Figure above explains how the code works in our implementation. Details on each activity in the code flow is given below:
Details on each activity in the code flow is given below:
CVE Dictionary: Mapping of a list of CVE Numbers for the Metasploit Modules.
Rule Set: Mapping of a list of Snort Rules to the Metasploit Module.
Vulnerability Dictionary: Mapping of a list of Metasploit Modules for a Target OS.
Writing Snort rule to file:
- For each module in the Vulnerability Dictionary for the OS, we write the rules on to the local.rules file which will be used by Snort for detection of malicious traffic.
The code responsible for this is given in write_to_snort.
Linking Action and Target version:
- For each CVE in the Vulnerability dictionary for the OS, we call the function to fetch the snort rule for the corresponding CVE for the module. A list of rules is returned for each Target OS.
The code responsible for this is given in vulnerability_mapper.
Fetching Snort Rule for each CVE:
- For each CVE given to the function, we first iterate through the rule set and see if there is a rule for the CVE in the keys. If there is one, we append this to a list which we will then return.
The code responsible for this is given in snort_gen.
Creation of dictionary mapping CVE to Exploit and exploit to target:
-
This is the function responsible for creating the vulnerability dictionary as well as the CVE Dictionary.
-
The first step in this function is to discover all the good, average and excellent metasploit modules for a given Network Service (using the description keyword) or for a target OS and platform.The command for achieving that is as follows:
msfconsole -q -x \"search type:exploit platform:<platform> target:<target> description:<service> rank:excellent rank:good rank:average\" -
The output from the command is parsed and the exploit names are extracted for each command. With the exploit names, we then use the “info” command to identify the potential CVEs linked to each module. Please note that there may be cases of modules which do not have the CVEs linked. These will be cast as part of the Modules which we are unable to find a Snort rule as part of the Community set or Earlier releases.
-
The list of vulnerabilities which have been inferred from the command are returned while the CVE dictionary mapping each Exploit/Module to its CVE is created.
The code responsible for this is given in create_vulnerabilty_dict.
Creating rule for each CVE from Community Ruleset and modifying to suit test topology:
-
First we iterate through each of the Exploits/Modules for the target OS (Vulnerability Dictionary).
-
Then we iterate through each of the CVEs for the Exploit/Module in the CVE Dictionary.
-
With each CVE, we iterate through the Community Ruleset to do a regex match with the CVE part of the regex search string. For each CVE which matches the search, modifications with the HOMENET and EXTERNALNET are made to suit the current test topology.
-
This rule is then appended to the rule set which contains the exploit name as key and the list of rules for that exploit as value.
The code responsible for this is given in create_rule_set.
Creating rule for each CVE from earlier versions of the community ruleset and modifying to suit test topology:
-
The same steps are followed as previously but with a list of files from earlier releases replacing the single current release of the Community Ruleset.
-
With each CVE, instead of iterating through the Community Ruleset, we iterate through a list of files which are previous community releases of the rule set to do a regex match with the CVE part of the regex search string. For each CVE which matches the search, modifications with the HOMENET and EXTERNALNET are made to suit the current test topology.
-
This rule is then appended to the rule set which contains the exploit name as key and the list of rules for that exploit as value.
The code responsible for this is given in create_rule_set_github.
Helper function to calculate coverage percentage of Ruleset:
- This helper function just calculates the coverage percentage by simply finding out the number of exploits in the Rule Set which do not have a Snort Rule as part of the Values and divides this number by the total number of exploits for the particular target OS/network service returned as output of search command.
The code responsible for this is given in coverage_count.
Technical Challenges
-
Infrastructure Infrastructure was a challenge.During our initial phase of the project, we decided to deploy use Amazon AWS, but unfortunately it doesn’t support vulnerable software versions. Eventually we decided to make a setup of test-bed using our local machines but that wasn’t feasible either due to resource crunch.
-
Attack surface Coverage Snort community does not have an exhaustive list of CVEs and their snort rules. Including earlier releases of the community rulesets (git repository) increased the rule coverage however, this does not cover all possible vulnerabilities.
-
Extra rules even if system is actually not vulnerable Our algorithm puts in snort rules for all vulnerabilities that are possible in the OS. It is possible that the vulnerability is introduced only when a particular software is installed on the OS however, snort rules for this vulnerability will be be included.
-
Metasploit CVE Coverage Some of the Metasploit modules are not linked to any CVEs . Hence, it is impossible to find rules in either the earlier releases of the community ruleset (git repository) or in the current release of the community ruleset.
-
Innocous Traffic Segregation in PCAPs Currently we are only able to isolate any injection attacks by the Metasploit Modules and help craft Snort Rules to alert against those attacks.
Evaluation Setup The figure of the setup depicts our testbed setup. We have a single attacker VM and multiple vulnerable VMs in our setup. The attacker VM has network connectivity to the victim machines. Snort is in the middle of the attacker and victims, monitoring egress traffic from attacker VM. Snort variables “HOMENET” and “EXTERNALNET” are modified to suit the current test topology.
We have deployed multiple victim OS such as Windows Server 2008, 2012 and 2016 R2 versions. We have explored vulnerabilities on common network protocols like SMB, UDP and HTTP. We tested them on some of most recent and in-famous vulnerabilities like “Eternal-Blue”.
Our metrics of interest are to ensure a that when a known attack occurs, it gets detected successfully. Secondly, we expect a good snort rule coverage for known OS vulnerabilities. For this, we look for snort rules corresponding to CVEs in the “Community Rule-set” and Git repositories of earlier releases of the rule sets. We anticipate to analyze the packet captures and design the SNORT rules based on attack packet payload if snort rules are not found with previous approaches.
Methodology The Snort website has list of community rule-sets which was initially used to construct OS to rules dictionary. The community rule-set is a collection of frequently used Snort rules for different CVEs. Preliminary results in this stage with this rule-set gave us an accuracy of 15% i.e. 15% of the exploits had a direct mapping of a CVE listed in the references of the snort rule. Further including earlier versions of the rulesets available in github repositories, we were able to achieve an increased hit percentage of 70%.
For the purpose of testing our system, we use Metasploit to conduct the attack. Here we are assuming that the attacker is aware of the operating system of the victim machine. This can be found by running port scans by tools like nmap etc.
Using Metasploit, we as an attacker choose the SMB protocol and choose to target Windows 2008 R2 Server machine. The EternalBlue attack is simulated in the following way-
-
Set the exploit as “windows/smb/ms17/eternalblue”
-
Set the payload to be meterpreter reverse TCP session
-
Set the required options by configuring LHOST(local host) as attacking machine and RHOST(victim machine or remote host)
-
Finally execute the attack using “exploit” command
As shown in Figure 4 we have 2 terminals open. One is the attacking terminal and the other terminal depicts the output of the attack. As we execute the above mentioned steps, we observe that the Snort was able to detect and report the attack successfully.
How the algorithm works against the attack:
-
Code gets OS version Windows 2008 R2 as input.
-
The code first discovers all the good, average and excellent metasploit modules for a target OS and platform using “search” command :
msfconsole -q -x search type:exploit platform:Windows target:Windows 2008 R2 rank:excellent rank:good rank:average.
“info” command is to identify the potential CVEs linked to each module.
-
Corresponding snort rules are found in Snort community and added in local.rules file.
By integrating Metasploit and Snort with our script we are able to capture the vulnerabilities in a device newly added into the network and also detect if an attack happens on the network devices.
As of now, the algorithm depends on snort community rule set and git repositories to find snort rules for CVEs in Metasploit. While these tools have 60-70% coverage on finding a rule set for the vulnerability , we are aware that not all CVEs are in snort community base. Having the ability to add git repositories allows us to easily increase our knowledge base of snort rules with a caveat of having to trust the git repositories to be reliable. In future we can look at PCAP of incoming traffic to identify an attack. For example, currently a code injection attack based on Buffer Overflow could be carried on port 4444 and traffic to this port can be flagged. We anticipate to achieve greater coverage with this approach.
This project is being advised by Matthew McCormack, a PhD student working with Professor Vyas Sekar as part of a bigger problem titled “Automatically adding network defenses”.
def write_to_snort(version, rules):
f = open("/etc/snort/local_test.rules", "a+")
for i, CVE_rules in enumerate(rules):
f.write("# "+vulnerability_dict[version][i]+"\n")
for rule in CVE_rules:
f.write(rule + "\n")
f.write("\n")
f.close()
def vulnerability_mapper(action, version):
res = []
if version in vulnerability_dict.keys():
for CVE in vulnerability_dict[version]:
res.append(snort_gen(action, CVE))
return res
def snort_gen(action, CVE):
return_list = []
if CVE in rule_set.keys():
for item in rule_set[CVE]:
return_list.append(action + " " + item)
return return_list
def create_vulnerabilty_dict(platform, target, module):
if platform != ' ':
if target != ' ':
if module != ' ':
child = pexpect.spawn('msfconsole -q -x \"search type:exploit platform:'+platform+ ' target:'+target+' description:'+module+' rank:excellent rank:good rank:average\"')
else:
child = pexpect.spawn('msfconsole -q -x \"search type:exploit platform:'+platform+ ' target:'+target+' rank:excellent rank:good rank:average\"')
else:
if module != ' ':
child = pexpect.spawn('msfconsole -q -x \"search type:exploit platform:'+platform+ ' description:'+module+' rank:excellent rank:good rank:average\"')
else:
child = pexpect.spawn('msfconsole -q -x \"search type:exploit platform:'+platform+ ' rank:excellent rank:good rank:average\"')
else:
if target != ' ':
if module != ' ':
child = pexpect.spawn('msfconsole -q -x \"search type:exploit target:'+target+' description:'+module+' rank:excellent rank:good rank:average\"')
else:
child = pexpect.spawn('msfconsole -q -x \"search type:exploit target:'+target+' rank:excellent rank:good rank:average\"')
else:
if module != ' ':
print("msfconsole -q -x \"search type:exploit description:"+module+" rank:excellent rank:good rank:average\"")
child = pexpect.spawn('msfconsole -q -x \"search type:exploit description:'+module+' rank:excellent rank:good rank:average\"')
else:
child = pexpect.spawn('msfconsole -q -x \"search type:exploit rank:excellent rank:good rank:average\"')
child.expect('msf5')
cmd_show_data = child.before
output = cmd_show_data.decode('utf-8').split('\r\n')
output = output[15:]
output = list(filter(('').__ne__, output))
vulnerabilities = []
for data in output:
found = re.findall(r'exploit/\w*/\w*/(\w*)',data)
entire_vuln = re.findall(r'(exploit/\w*/\w*/\w*)',data)
if found:
vulnerabilities.append(found[0])
child.sendline('info '+entire_vuln[0])
child.expect('msf5')
reverse_dict[found[0]] = entire_vuln[0]
cve_number = child.before
cve_list = []
for line in cve_number.decode('utf-8').split('\r\n'):
if re.findall(r'cve/(\w*-\w*-\w*)/', line):
cve_list.append(re.findall(r'cve/(\w*-\w*-\w*)/', line)[0])
cve_dict[found[0]] = cve_list
child.sendline('exit')
return vulnerabilities
Mapping rule for each CVE from Community ruleset and modifying this mapped rule to suit test topology
def create_rule_set(target):
print("Creating Rule Sets for the vulnerabilities from the Snort Rule Set")
fline = open("/root/Downloads/snort3-community-rules/snort3-community.rules", "r").readlines()
if target in vulnerability_dict.keys():
for vuln_module in vulnerability_dict[target]:
if vuln_module in cve_dict.keys():
if vuln_module not in rule_set.keys():
rule_set[vuln_module] = []
for cve in cve_dict[vuln_module]:
for line in fline:
if '-'.join(cve.split('-')[1:]) in line or ','+cve.split('-')[2] in line or '/'+cve.split('-')[2] in line:
line = line.replace('# ','')
if line.split(' ', 1)[1].replace("$HOME_NET", "any").replace("$EXTERNAL_NET", "any") not in rule_set[vuln_module]:
rule_set[vuln_module].append(line.split(' ', 1)[1].replace("$HOME_NET", "any").replace("$EXTERNAL_NET", "any"))
# print(json.dumps(rule_set, indent = 1))
Mapping rule for each CVE from earlier releases of the ruleset and modifying this mapped rule to suit test topology
def create_rule_set_github_repo(target):
print("Creating Rule Sets for the vulnerabilities from the Custom Github Repositories")
for filename in only_files:
# print("Going through : "+filename)
fline = open("/root/Downloads/snort-rules/snortrules-snapshot-3000/rules/"+filename, "r").readlines()
if target in vulnerability_dict.keys():
for vuln_module in vulnerability_dict[target]:
if rule_set[vuln_module] == []:
for cve in cve_dict[vuln_module]:
for line in fline:
if '-'.join(cve.split('-')[1:]) in line or ','+cve.split('-')[2] in line or '/'+cve.split('-')[2] in line:
line = line.replace('# ','')
if line.split(' ', 1)[1].replace("$HOME_NET", "any").replace("$EXTERNAL_NET", "any") not in rule_set[vuln_module]:
rule_set[vuln_module].append(line.split(' ', 1)[1].replace("$HOME_NET", "any").replace("$EXTERNAL_NET", "any"))
def read_pcap(filename):
a = rdpcap(filename)
f = open("/etc/snort/local_test.rules", "a+")
sessions = a.sessions()
for session in sessions:
http_payload = ""
cmd = ""
for packet in sessions[session]:
try:
if packet[TCP].sport == 4444 and packet[TCP].payload:
hexd = binascii.hexlify(bytes(packet[TCP].payload))
cmd = "alert tcp any "+str(packet[TCP].sport)+" any any (msg: \"Possible attack on port 4444\"; payload: "+str(hexd)+"; classtype: attempted-admin; reference: url, github.com/ptresearch/AttackDetection; metadata: Open Ptsecurity.com ruleset; sid: 120020; rev: 3;)\n"
elif packet[TCP].dport == 4444 and packet[TCP].payload:
hexd = binascii.hexlify(bytes(packet[TCP].payload))
cmd = "alert tcp any any any "+str(packet[TCP].dport)+" (msg: \"Possible attack on port 4444\"; payload: "+str(hexd)+"; classtype: attempted-admin; reference: url, github.com/ptresearch/AttackDetection; metadata: Open Ptsecurity.com ruleset; sid: 120020; rev: 3;)\n"
f.write(str(cmd))
except Exception as e:
print(e)
f.close()
def coverage_count(dictionary):
empty = total = 0
for item in dictionary.keys():
if dictionary[item] == []:
empty += 1
empty_list.append(reverse_dict[item])
total += 1
empty_percent = (empty / total) * 100
print("Total number of exploits in the search command", total)
print("Total number of exploits which have a Snort Rule in the Community Ruleset", (total - empty))
print("Total number of exploits which do not have a Snort Rule in the Community Ruleset", (empty))
return empty_percent