CRITs is a web-based tool which combines an analytic engine with a cyber threat database that not only serves as a repository for attack data and malware, but also provides analysts with a powerful platform for conducting malware analyses, correlating malware, and for targeting data. These analyses and correlations can also be saved and exploited within CRITs. CRITs employs a simple but very useful hierarchy to structure cyber threat information. This structure gives analyst the power to 'pivot' on metadata to discover previously unknown related content.
Visit our website for more information, documentation, and links to community content such as our mailing lists and IRC channel.
CRITs is designed to work on a 64-bit architecture of Ubuntu or RHEL6 using Python 2.7. It is also possible to install CRITs on CentOS and OSX. There is more information in the 'documentation' folder if you would like to see how you can install CRITs on OSX. These instructions are mainly for developers as OSX is not a supported operating system.
If you require the use of a 32-bit OS, you will need to download 32-bit versions of the pre-compiled dependencies.
The following instructions assume you are running Ubuntu or RHEL6 64-bit with Python 2.7. If you are on RHEL which does not come with Python 2.7, you will need to install it. If you do, ensure all python library dependencies are installed using Python 2.7. Also, make sure you install mod_wsgi against the Python 2.7 install. More information on this can be found in the Github wiki at https://github.com/crits/crits/wiki/Common-Questions.
CRITs has a decent amount of dependencies due to the extensive amount of functionality provided throughout the system.
You can find a repository of the existing dependencies here. Read the README that comes with the dependencies for information on how to use the install script.
During heavy volumes of data input or usage, the system can create a lot of network traffic. We recommend doing some server tuning to improve performance. Execute the following to decrease the amount of TCP connections piling up during batch operations:
echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse
echo 1 > /proc/sys/net/ipv4/tcp_tw_recycleEnsure these values are set on reboot by adding the above commands to /etc/rc.local or whatever comparable config in your Linux distro.
If you would like to extend the functionality of CRITs by adding services, you will have to follow these steps:
You can find some readily available community-developed services here.
Put the services anywhere you would like on your server (you can even have multiple directories). With the repository above you could use something like /data/crits_services as your directory.
You can configure CRITs with the location(s) of the directory or directories later on using the setconfig management command (explained below) or in the CRITs Control Panel UI (explained below).
CRITs uses MongoDB to store authentication, session, metadata and binary information. MongoDB is a non-relational database. If you would like to read more about MongoDB and how to administer it, here are some links:
Most people starting out will run a single-server MongoDB instance. If you are unsure as to whether or not you should run a single instance or a cluster, here is some information:
- Can run with a couple CPUs, and lots of RAM/Disk.
- minimum of 8-16G of RAM.
- 10G of disk dedicated to MongoDB.
- Good for 10's of thousands of binaries, emails, indicators, and PCAPs
- Long term horizontal and vertical scaling.
- Recommend using 1U servers.
- RAM and disk take priority over CPU.
- Recommend maxing RAM per box (64G+).
- Local storage is by far the preference here (500G+).
- Good for millions of binaries, emails, indicators, and PCAPs
- As you need more space, you can quickly and easily add servers.
If you are looking to setup a cluster, please read and follow the information in the 'mongo_cluster.txt' file which can be found in the 'documentation' directory.
- Create the database directory:
sudo mkdir -p /data/db- In the 'contrib' directory that came with CRITs, you will find a mongo directory with two directories in it: one for Ubuntu, and one for RHEL. They contain start scripts for your mongo processes. These scripts properly configure reclaim_mode on your server and start the mongod process.
cdto the directory for your OS and run the mongod_start.sh script:
sudo ./mongod_start.sh- Verify this is working by connecting to it with the following command:
mongoThis should bring up the mongo shell on localhost
Apache is generally the preferred web server for production instances of CRITs. If you are running a development instance, skip down to the section below titled Installing CRITs using the Django runserver.
- Create a crits user on your system:
adduser crits- Modify the crits group to contain your webserver user, any users running CRITs cronjobs, and any users running CRITs scripts.
- Create the CRITs directory structure:
sudo mkdir -p /data/crits- Put CRITs code in /data/crits then cd to it
- Change the group owner of the logs directory and the logs/crits.log file to 'crits'.
chgrp -R crits logs- The 'logs/crits.log' file needs permissions of 664.
touch logs/crits.log
chmod 664 logs/crits.logIn the crits/config directory that came with the CRITs codebase, copy database_example.py to database.py:
cp database_example.py database.pyEdit database.py and use the comments to configure your MongoDB connection information and your SECRET_KEY. If you are unsure what S3 is or if you are using it, leave FILE_DB alone.
NOTE: at this point you should have MongoDB running!
Run the create_default_collections management command to setup your database:
python manage.py create_default_collectionsIf you have issues running this command, it is usually because of a few things:
- There is a missing dependency. Go back through the dependency installation and make sure everything was installed properly.
- You did not set your SECRET_KEY in the previous step.
Take a look at the options for the users management command:
python manage.py users -hUse that command to setup your first admin user for CRITs. Be sure to use -A to set them as an admin. Make note of the temporary password provided in the output!!
Django needs to know the host(s) or domain name(s) that you will be serving your CRITs instance from for security purposes. To set this, run the following command:
python manage.py setconfig allowed_hosts "foo"Where "foo" is the host/domain name, or a comma separated list of names that will be serving CRITs. For more information on this, please see the Django Reference.
If you'd like to configure CRITs prior to starting the webserver, check out the setconfig management command:
python manage.py setconfig -hIf you'd rather the system start with sane defaults that you can modify via the web interface, you can proceed to Starting the UI. NOTE: Some config option changes might require a web server restart!
We have instructions for setting up Apache on Ubuntu and RHEL. Each provides instructions for generating a temporary SSL certificate if you do not have an official one (yet!).
For installs on a new Apache instance you can follow the steps below. For existing Apache installations, use the files mentioned as a guideline.
- Stop Apache:
sudo /etc/init.d/apache2 stop- In /etc/apache2/, remove sites-available:
sudo rm -rf /etc/apache2/sites-available- From the extras folder that came with the CRITs codebase:
sudo cp *.conf /etc/apache2
sudo cp -r sites-available /etc/apache2- In /etc/apache2/sites-enabled, remove the default, and link to /etc/apache2/sites-availble/default-ssl instead:
sudo rm /etc/apache2/sites-enabled/*
sudo ln -s /etc/apache2/sites-available/default-ssl /etc/apache2/sites-enabled/default-ssl- Generate self-signed cert (can and should be replaced by an official cert from a trusted source):
cd /tmp
sudo openssl req -new > new.cert.csr
sudo openssl rsa -in privkey.pem -out new.cert.key
sudo openssl x509 -in new.cert.csr -out new.cert.cert -req -signkey new.cert.key -days 1825
sudo cp new.cert.cert /etc/ssl/certs/crits.crt
sudo cp new.cert.key /etc/ssl/private/crits.plain.key- Enable SSL support in Apache2
sudo a2enmod ssl- Ensure that LANG environment variable is set to en_US-UTF.8 in /etc/apache2/envvars:
export LANG=en_US.UTF-8
- Start apache2
sudo /etc/init.d/apache2 startFor installs on a new Apache instance you can follow the steps below. For existing Apache installations, use the files mentioned as a guideline. Depending on the version of Apache you may need to adjust some config values. See documentation about Django, mod_wsgi, and Apache on the Django website.
- Stop Apache:
/etc/init.d/httpd stop- From the extras folder of the CRITs codebase, copy the rhel_httpd.conf file to /etc/httpd/conf folder:
sudo cp rhel_httpd.conf /etc/httpd/conf/httpd.conf- From the extras folder of the CRITs codebase, copy the rhel_ssl.conf file to /etc/httpd/conf.d folder:
sudo cp rhel_ssl.conf /etc/httpd/conf.d/ssl.conf- Generate self-signed cert (can and should be replaced by an official cert from a trusted source):
cd /tmp
sudo openssl req -new > new.cert.csr
sudo openssl rsa -in privkey.pem -out new.cert.key
sudo openssl x509 -in new.cert.csr -out new.cert.cert -req -signkey new.cert.key -days 1825
sudo cp new.cert.cert /etc/pki/tls/certs/crits.crt
sudo cp new.cert.key /etc/pki/tls/private/crits.plain.key- Start apache2
sudo /etc/init.d/httpd startMake sure you can only get to https://your-site.com/ and not http://your-site.com/
Proceed to the Final Steps section for more information on cronjobs and other useful things.
The Django runserver is our recommended web server for development or test instances of CRITs. It is quick, light, and provides a way for developers and administrators to look at the web server requests/responses in real time. It is also useful for debugging and viewing print statements.
NOTE: This configuration does not use SSL. If you would like to use runserver over SSL you can read about it here
If you are a developer cloning a git repository, we generally recommend you clone to ~/git/crits. If you are using a release tarball, explode the tarball in the place of your choosing.
In the crits/config directory that came with the CRITs codebase, copy database_example.py to database.py:
cp database_example.py database.pyEdit database.py using the comments to configure your MongoDB connection information and your SECRET_KEY. If you are unsure what S3 is or if you are using it, leave FILE_DB alone.
NOTE: at this point you should have MongoDB running!
Run the create_default_collections management command to setup your database:
python manage.py create_default_collectionsIf you have issues running this command, it is usually because of a few things:
- There is a missing dependency. Go back through the dependency installation and make sure everything was installed properly.
- You did not set your SECRET_KEY in the previous step.
Take a look at the options for the users management command:
python manage.py users -hUse that command to setup your first admin user for CRITs. Be sure to use -A to set them as an admin. Make note of the temporary password provided in the output!!
Django needs to know the host(s) or domain name(s) that you will be serving your CRITs instance from for security purposes. To set this, run the following command:
python manage.py setconfig allowed_hosts "foo"Where "foo" is the host/domain name, or a comma separated list of names that will be serving CRITs. For more information on this, please see the Django Reference.
If you'd like to configure CRITs prior to starting the webserver, check out the setconfig management command:
python manage.py setconfig -hIf you'd rather the system start with sane defaults that you can modify via the web interface, you can proceed to Starting the UI. NOTE: Some config option changes might require a web server restart!
Run the following (change IP and port appropriately) to start the Django runserver:
python manage.py runserver 1.2.3.4:8080At this point you should be able to point your browser to that location and see the CRITs login screen. Proceed to the 'Final Steps' section for more information on cronjobs and other useful things.
The main cronjob we recommend is for the script which executes common mapreduce jobs. These jobs do things like collect database statistics, generate Campaign information, and other useful bits of information. If you would like the Counts and stats updated on your Dashboard, you will need to add this.
We also support sending batch email notifications to users of your system. The email provided a non-detailed overview of how many changes have happened to items they are subscribed to. This cronjob also updates the notifications users will see in the interface.
As a user who has access to the codebase and to execute python code, edit their crontab:
crontab -eAdd the following entries, making adjustments for the folder path and the frequency you want them to run:
0 * * * * cd /data/crits/ && /usr/bin/python manage.py mapreduces
0 * * * * cd /data/crits/ && /usr/bin/python manage.py generate_notificationsIn most cases this step is not neccessary, but if you do have any custom django settings that you need to change for your local environment, In the crits/config directory you can copy overrides_example.py to overrides.py. You can look at the example for some ideas on what you can do with this. This file gets included at the end of the settings.py, so you can do pretty much anything you would like there, but this allows you to isolate your changes from the settings.py distributed in the package.
If you opted to configure CRITs after the web server is up, or want to change and values after it is up, you can go to the CRITs Control Panel in the navigation menu as an admin. From there, you can make your changes.
For more information, check the FAQ in the documentation directory and the Github wiki page. If you’d like to generate Sphinx documentation for the code, you can run make html in the documentation/src directory. There is also a Help guide in the UI.
Thanks for using CRITs!