Long term log storage extension for OSSIM (Open Source Security Incident Management), AlienVault
The basic architecture of the system can be seen on ossim-logengine_basic_architecture.jpg. The diagram shows the optimal case, where the main Ossim and the logarchiver are on different computers. The Ossim system is presumed to contain, at least, the base Ossim runtime, with a webserver, and the dashboard included, and also a MySQL database, containing the Ossim event records. The logserver component is set up to contain the Ossim logserver component, including a Tornado based web server, and the archiver scripts, and also a MongoDB runtime, to hold the collected data. As ossim-logengine_basic_architecture.jpg shows, the main communication between the two systems takes place through two component. First, the archiver on the logserver side queries the MySQL found at the Ossim side. The queried records are then reformatted, parsed, and stored in the MongoDB. The second connection is through the web servers. The Ossim dashboard loads the custom plugin, which grants access to the logarchive. The system is set up in a way, that the Tornado server on the logserver side only accepts queries from the specific dashboard page, created for this use-case. The dashboard page queries an API endpoint found on the logserver side, which in response queries the MongoDB. The requested records are then returned to the dashboard page, where a dynamically working Angular page deserializes them from JSON, and displays them for human usage.
- tornado 4.3 for running the server and managing the file uploads.
- configobj 4.7.2 for parsing the configuration files.
- pymongo 4.7.2 for parsing the configuration files.
- pdfkit to generate pdf from python.
- wkhtmltopdf command line utility, pdfkit dependency.
- jinja2 HTML report generaton.
- pprint used in report generation.
- sqlalchemy 1.0.9 for querying the MySQL.
- sshtunnel 0.0.8.2 for creating an sshtunnel to the MySQL server.
- pymongo 3.2.2 for interacting with MongoDB.
Installing ossim-logarchiver part 1 (ossim-additions)
-
Log in to OSSIM
-
select "Jailbreak System" from the menu, then Yes
-
Optional time synchronization, two methods (ntpdate or tlsdate)
-
copy ossim-additions folder to /opt
-
cd /opt/ossim-additions/
-
edit install.sh, put server IP and port (default 4343)
-
run install.sh
-
copy/save the generated password! it will be needed later
Installing ossim-logarchiver part 2 (ossim-logserver)
- Copy the ossim-logserver folder to a fresh installed Debian Linux (7-8)
- edit config/etc/ossim/logserver.conf The following configuration values can be changed:
- loglevel - how verbose the log output should be (DEBUG, INFO, WARNING, ERROR)
- logfile - the file where the logging output is redirected to - make sure archiveuser can write this file!
- archive_directory - the directory where the archived files are put into - make sure archiveuser can write to this directory!
- server (this part should not be modified)
- port - the port where the Tornado webserver listens *_callback_time - the frequency of archiving operations (in seconds) mongo (in general it should not be modified)
- database - the name of the database where the archiving happens
- hostname - the hostname / IP of the mongoDB server
- port - the port on which the mongoDB instance will be running - collection - the name of the collection inside the mongoDB database
- justinsert and justread - dynamically generated, please do not change (setup script generates them) mysql password - this is dynamically generated, please do not change (setup script generates it)
- hostname - the hostname / IP of the AlienVault OSSIM server (in the example: 192.168.2.1)
- port - the SSH port of the AlienVault OSSIM server (in the example: 22)
- database - the database where OSSIM stores the events
- username - the username that will be created in the MySQL DB
- tunnel_pwd - the generated password of the tunnel user
- this was generated by the ossim-additions install script
There are additional configuration parameters at the top of install.sh that can be set. The most important is EMAIL, which defines an e-mail address where a notification will be sent to in case any important services that ossim-logserver relies on stop. Two more configuration parameters should be set regarding the webservice configuration: the WEBPORT (default value: 4343), which defines the port where ossim-logserver will be serving requests, and the SERVERNAME, which defines the Host/Server name where the logserver will be available.
- Run install.sh
When installing on an OSSIM host, you might get a warning about some apt repositories not being available and pip might throw some errors about not finding some directories. The script will ask you for input for the last time, in the following order: desired MongoDB root user and password, the MySQL (AlienVault OSSIM) root user (default: root) and password (it can be found in /etc/ossim/ossim_setup.conf in the database section), whether the MySQL is hosted on the localhost or not (default: not) SSH (AlienVault OSSIM) root (NOT TUNNEL!, default: root) user and password. The SSH root login is required to create a MySQL user on that server. An additional database (logarchive) and a table (new_ids) within that will be created. An INSERT trigger will also be created on the alienvault_siem database's acid_event table, which will copy the newly inserted events' IDs to the new_ids table. During the installation, a root CA plus intermediate certificate pair is created to serve as the HTTPS cert for the server. Please take care not to use the same O (organization name) for both root and intermediate certs and to use the correct CN (common name) (typically the IP of the server as seen from the OSSIM AlienVault server). Ifyouwishtouseyourowncertificates, placetheminside /etc/nginx/ssl as server.crt (certificate)and server.key (privatekey). Ifyouchoosetoinstallyourowncertificates, do not forget to generate the Difie-Hellman parameters with openssl dhparam -dsaparam -out /etc/nginx/ssl/dhparam.pem 4096.
After the installation, do not forget to install the certificate of the server (the root CA's) on all other servers which will access the log archiver. You can install the certificates (on Debian-based machines) by issuing the following commands as a privileged user: Skip the following line if installing on the same server as OSSIM (LOGSERVER) $ scp /root/ssl/rootCA.pem root@192.168.2.1:/root/ (OSSIM)$ cp rootCA.pem /usr/local/share/ca-certificates/server.crt (OSSIM)$ update-ca-certificates
If you encounter problems during operation you should:
- check top or htop on Logserver machine
- check /var/log/ossim/logserver.log on Logserver machine
- check /var/log/mongodb on Logserver machine
- check df -h on Logserver machine
- check top or htop on Ossim machine
- wget https://logserver-ip/status from Ossim machine
- check /var/log/mysql* logs on Ossim machine