THIS PROJECT IS DEPRICATED, PLEASE USE FENNEC INSTED
osqueryIR is an artifact collection tool for Linux systems. It provides the following capabilities:
- Execute osquery SQL queries
- Collect files and folder
- Execute system commands
- Parse log files (ex. nginx, auth, syslog, etc) using regex
-
Clone this repo
git clone https://github.com/abdulrhmanalfaifi/osqueryIR
-
Download python dependencies
python3 -m pip install -r requirments.txt
-
Try it using this command
python3 osqueryIR.py -h
The following is the help message for osqueryIR:
usage: osqueryIR.py [-h] [--osquery-binary OSQUERY_BINARY] [-c CONFIG]
[-o OUTPUT] [-q] [--log-file-name LOG_FILE_NAME]
[--log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}]
[--output-format {jsonl,kjson}] [--disable-collect]
A Linux artifact collection tool
optional arguments:
-h, --help show this help message and exit
--osquery-binary OSQUERY_BINARY
osqueryd binary path (Default=./osqueryd)
-c CONFIG, --config CONFIG
Path to the configuration file (Default=./config.yaml)
-o OUTPUT, --output OUTPUT
Change the output folder name (Defaults to the machine
hostname)
-q, --quiet Do not print log messages
--log-file-name LOG_FILE_NAME
Name of the log file (Default=osqueryIR_log)
--log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}
Set logging level (Default=INFO)
--output-format {jsonl,kjson}
Change the output format (Default=jsonl)
--disable-collect Disable collection artifacts
-
--osquery-binary
: osqueryd binary path. By default it will uses the binary in this repo. -
-c
or--config
: path to osqueryIR configuration. By default it will beconfig.yaml
in this repo. -
-o
or--output
: the output file (zip file) name, By default it will be the machine hostname. -
-q
or--quit
: Do not print logging to the stdout. osqueryIR will always write the log to the output file. -
--log-file-name
: change the default name for the log file (osqueryIR_log). -
--log-level
: the log level, default isINFO
. -
--output-format
: osqueryIR support writing the results in two different formats:- jsonl: a newline separated JSON object. Each object represent a record.
- kjson: the format understood by Kuiper. If you are planing to use Kuiper for analysis then you should use this format.
-
--disable-collect
: disable artifact collecting. Only parsing and osquery artifacts will be acquired.
osqueryIR accepts a configuration file that contains artifact specification. The following is an example configuration along with comments:
artifacts:
# Name of artifact. the results will be saved to a file with this name
- logged_in_users:
# artifact type. queries run osquery SQL queries and return the results as json
queries:
- 'select * from logged_in_users'
# Optional: map the field called `name` to `@timestamp` and run the modfier `epoch_to_iso` on the value. `modifier` field is not required
maps:
- name: time
map_to: '@timestamp'
modifier: epoch_to_iso
# Optional: description of the artifact
description: 'Collect and parse the currently loggedin users'
- logs:
# artifact type. collect the specified files and directories without parsing
collect:
- '/var/log/**'
- '/home/*/.vnc/*.log'
description: 'Collect logs wellknow paths'
- auth_log:
# artifact type. parse the specified files using regex and return the results as json.
parse:
# files to parse
path: '/var/log/auth.log*'
# regex used for parsing
regex: '([A-Z][a-z]{2}[ ]{1,}[0-9]{1,2}[ ]{1,2}[0-9]{1,2}:[0-9]{2}:[0-9]{2}) ([a-zA-Z0-9_\-]+) ([a-zA-Z0-9_\-\]\(\)=\./]+)\[?([0-9]+)?\]?: (.*)'
# the name of the extracted fields
fields:
- 'time'
- 'hostname'
- 'service'
- 'pid'
- 'msg'
maps:
- name: time
map_to: '@timestamp'
modifier: time_without_year_to_iso
description: 'Parse auth logs from the path /var/log/, and return the results as jsonl/kjson'
- bad_logins:
# artifact type. Execute system command and return stdout & stderr
command:
- 'lastb'
To collect the artifacts from the provided configurations, execute the following command:
python3 osqueryIR.py
A file will be created named {HOSTNAME}.zip
that contains all artifacts.
osqueryIR can generate the result in kjson
format which could be ingested by Kuiper. To collect artifacts in kjson
format execute the following command:
python3 osqueryIR.py --output-format kjson --disable-collect
upload the file to Kuiper and execute the kjson
parser