User anomaly detector based on logs generated by Osquery framework and machine learning to process those logs. Machine learnings alogirthms that are currently implemented are: Recurrent neural network - Long short-term memory (LSTM) and One-Class Support Vector Machine (OCSVM). This project is part of Master thesis at Faculty of electrical engineering and computing, University of Zagreb.
Note: Only Linux platforms are supported. Tested on Xubuntu 16.04.
The code includes:
- dataset for different users to train and test models
- preprocessing functions to properly tokenize the data
- train and predict functions for existing (saved) data
- training new model on raw osquery logs
- saving and loading trained models
- predict functions for incoming new data
Existing security solutions are mostly based on preventing known malicious threats or a defined set of rules and therefore most outside and inside threats end as successful attacks. The idea was to build a system that is an adaptive user action identifier, so it can predict and detect anomalous behavior in real time.
- Download osquery.
- Move osquery pack of queries file
user-behavior.confand osquery configuration fileosquery.conffrom conf directory to osquery directory which is usually/etc/osquery/or/usr/local/. - To capture syslog events additional configuration is required.
(Ubuntu) Add the following to your rsyslog configuration files (usually located in
/etc/rsyslog.confor/etc/rsyslog.d/:
template(
name="OsqueryCsvFormat"
type="string"
string="%timestamp:::date-rfc3339,csv%,%hostname:::csv%,%syslogseverity:::csv%,%syslogfacility-text:::csv%,%syslogtag:::csv%,%msg:::csv%\n"
)
*.* action(type="ompipe" Pipe="/var/osquery/syslog_pipe" template="OsqueryCsvFormat")
If you use syslog-ng or you can read about the configuration here. If no logs are available, read debugging suggestions.
- Download the repository using
git clone. - Install pip requirements with the following command:
pip install -r requirements.txt. Python 2.7 is required. I don't guarantee that everything works with Python 3+ but please feel free to try.
- Start osquery first i.e.
sudo service osqueryd startor/usr/local/bin/osqueryd. Osquery in this case doesn't require root access. - You can start the program by choosing the wanted algorithms i.e.
python main.py -a OCSVM. Default algorithm is LSTM. You can also change the queries result log file using the-lflag, default is/var/log/osquery/osqueryd.results.log. If you are running osquery as root, use sudo to run python script because it needs to be able to read log file. - Training model is built for each user since each user is expected to have different behavior. First part of the main script is training a new model by processing all actions in the log file and considering them as normal behavior. On top of that model, predictions are made on each new action that comes in.
The MIT License Copyright (c) 2017-present