Skip to content

Zeek Package that extracts features from IRC communication

License

Notifications You must be signed in to change notification settings

stratosphereips/zeek-package-IRC

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

42 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

zeek-package-IRC

IRC Feature Extractor Zeek Package extends the functionality of Zeek network analysis framework. This package automatically recognizes IRC communication in a packet capture (pcap) file and automatically extract features from it. The goal for the feature extraction is to describe an individual IRC communications that occur in the pcap file as accurately as possible.

Installation

To install the package using Zeek Package Manager, run the following command:

$ zkg install zeek/stratosphereips/zeek-package-IRC

Run

To extract the IRC features on the selected pcap file that contains IRC, run the following command in a terminal:

$ zeek IRC-Zeek-package -r file.pcap 

The output will be stored in irc_features.log file in zeek log format. The log will look like this:

#separator \x09
#set_separator	,
#empty_field	(empty)
#unset_field	-
#path	irc_features
#open	2020-01-27-21-54-41
#fields	src	src_ip	src_ports_count	dst	dst_ip	dst_port	start_time	end_time	duration	msg_count	size_total	periodicity	spec_chars_username_mean	spec_chars_msg_mean	msg_word_entropy
#types	string	addr	count	string	addr	port	time	time	double	count	int	double	double	double	double
T!T@null	192.168.100.103	4	#a925d765	111.230.241.23	2407	1532322898.819018	1534860900.996931	2538002.177913	23	48705	0.050294	0.25	0.908075	2.001506
T!T@null	192.168.100.103	33	#a925d765	185.61.149.22	2407	1530166710.153128	1535620500.535362	5453790.382234	231	562890	1.0	0.25	0.908256	2.276218
#close	2020-01-27-21-54-41

Parsing log in Python

Instead of parsing the package manually, you can use the ZAT library in Python to parse it directly into a Pandas data frame or as a list of dictionaries.

There is an example of how to parse the log as a list of dictionaries:

import zat
from zat.log_to_dataframe import LogToDataFrame
from zat.bro_log_reader import BroLogReader


log_filename = 'irc_features.log'

logs_arr = []
reader = BroLogReader(log_filename)

for log in reader.readrows():
    # log is in dictionary format
    print(log)
    logs_arr.append(log) 

Description

Once the data was obtained from network traffic capture, there was a process to extract the features. We separated the whole pcap into communications for each individual user. To do that, we separated communication into the connections between the source IP, destination IP, and destination port (hereinafter IRC connection). The source port is randomly chosen from the unregistered port range, and that is why the source port is not the same when a new TCP connection is established between the same IP addresses. For this reason, we neglected the source port to match the IRC connection with the same source IP, destination IP, and destination port. This is shown in figure below, where are two connections from the source IP address (192.168.0.1), to the same destination IP address (192.168.0.2) using different source port.

alt

Example of IRC connection - IRC connection that is defined by source IP address 192.168.0.1, destination IP address 192.168.0.2, and destination port 440. Source port is neglected, and therefore one IRC connection can have multiple source ports. The IP addresses and ports are chosen randomly for demonstration purposes.

Extracted Features

The feature selection is made manually to provide a good means of characterizing malicious communication. Features were computed for each IRC connection. Here is a final list of features that we used in our models.

Total Packet Size

Size of all packets in bytes that were sent in IRC connection. It reflects how many messages were sent and how long they were.

Session Duration

Duration of IRC connection in milliseconds - i.e., the difference between the time of the last message and the first message in IRC connection.

Number of Messages

A total number of messages in IRC connection.

Number of Source Ports

As we have mentioned before, the source port is neglected in unifying communication into IRC connections because the it is randomly chosen when a TCP connection is established. We suppose that artificial users could have had a higher number of source ports than the real users since the number of connections of the artificial users was higher than the number of connections of the real users.

Message Periodicity

We suppose that artificial users (e.g., bots that are controlled by botnet master) use IRC for sending commands periodically, so we wanted to obtain that value. To do that, we created a method that would return a number between 0 and 1 - i.e. one if the message sequence is perfectly periodical, zero if the message sequence is not periodical at all.

To compute message periodicity, we firstly compute time differences between every message. On this computed sequence of numbers, we apply a fast Fourier transform (FFT). The output of FFT is a sequence of numbers. The higher the number on the given position of the output, the bigger the amplitude on the given position.Thus it has a more significant influence on the periodicity of the data. The position of the largest element in the FFT's output represents the length of the period, which is the most significant from all other periods.

To compute the quality of the most significant period, we split the data by length of that period.. Then we compute the normalised mean squared error (NMSE) that returns us the resulting number in the interval between 0 and 1 where 1 represents the perfectly periodic messages, and 0 represents not periodic messages at all.

Illustration of how message periodicity is computed. The time differences between messages and FFT output numbers are chosen randomly for demonstration purposes.

Message Word Entropy

To consider whether the user sends the same message multiple times in a row, or whether the message contains a limited number of words, we compute a word entropy across all of the messages in the IRC connection. By the term word entropy we mean a measure of words uncertainty in the message. For the computation of the word entropy, we use the formula below:

where n represents the number of words, and pi represents the probability that the word i will be used among all other words.

Username Special Characters Mean

We want to obtain whether the username of the user in the IRC communication is random generated or not. Therefore, in this feature, we compute the average usage of non-alphabetic characters in the username.

Message Special Characters Mean

With this feature, we obtain the average usage of non-alphabetic characters across all messages in the IRC connection. We apply the same procedure of matching special characters for each message as in the previous case - we match non-alphabetic characters by regex, and then we divide the number of matched characters by the total number of message characters. Finally, we compute an average of all the obtained values for each message.