A web-based tool to assist the work of the intuitive threat analysts.
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
compose
config
docs
manati
media
requirements
static
static1
utility
.coveragerc
.dockerignore
.editorconfig
.env-docker.example
.env.example
.gitattributes
.gitignore
.pylintrc
.travis.yml
CHANGELOG.rst
CONTRIBUTORS.txt
README.md
docker-compose.yml
manage.py
pytest.ini
requirements.txt
setup.cfg

README.md

ManaTI Project

Build Status contributions welcome GPLv3 license

Machine Learning for Threat Intuitive Analysis

The goal of the ManaTI project is to develop machine learning techniques to assist an intuitive threat analyst to speed the discovery of new security problems. The machine learning will contribute to the analysis by finding new relationships and inferences. The project will include the development of a web interface for the analyst to interact with the data and the machine learning output.

This project is partially supported by Cisco Systems. For more information about the project please go to Stratosphere Lab page

Stable Versions

  • Mon Sep 3 12:24:26 CEST 2018: Version 0.12.0a
  • Sun Aug 12 16:21:19 CEST 2018: Version 0.11.0a
  • Mon Jan 29 00:07:15 CEST 2018: Version 0.9.0a
  • Fri Nov 10 19:16:52 CEST 2017: Version 0.8.0.537a
  • Fri Mar 31 12:19:00 CEST 2017: Version 0.7.1
  • Sun Mar 5 00:04:41 CEST 2017: Version 0.7
  • Thu Nov 10 12:30:45 CEST 2016: Version 0.6.2.1
  • Wed Oct 12 21:19:21 CEST 2016: Version 0.5.1
  • Wed Sep 21 17:56:40 CEST 2016: Version 0.41
  • Tue Sep 13 10:52:36 CEST 2016: Version 0.4
  • Thu Aug 18 15:44:31 CEST 2016: Version 0.3
  • Wed Jun 29 10:44:15 CEST 2016: Version 0.2

Authors

App Screenshot

manati_screenshot

Installation

ManaTI is a Django project with a Postgres database and it works in Linux and MacOS. We recommend using a virtualenv environment to setup it. The installation steps for linux are:

    sudo apt-get update ; sudo apt-get upgrade -y
  1. Clone the repository
  2.     git clone git@github.com:stratosphereips/Manati.git; cd Manati
    

      or if you don't want to use SSH, use HTTPS

        git clone https://github.com/stratosphereips/Manati.git; cd Manati
    
  3. Install Virtualenv to isolate the required python libraries for ManaTI, also will be installed python libraries for development
  4.     sudo apt-get install virtualenv python-pip python-dev libpq-dev build-essential libssl-dev libffi-dev
    
  5. Create virtualenv folder
  6.     virtualenv .vmanati
    
  7. Active Virtualenv
  8.     source .vmanati/bin/activate
    
  9. Install PostgreSQL DB engine
  10.     sudo apt-get install postgresql-server-dev-all postgresql-9.5 postgresql-client-9.5
    
  11. Create environment variables files. Copy and rename the files .env.example to .env, and .env-docker.example to .env-docker
  12.     cp .env.example .env
        cp .env-docker.example .env-docker
    

    OPTIONAL

    You can modify the password and name of database, if you want. Remember, reflect the changes in the Postgres database settings below.

  13. Install required python libraries
  14.     pip install -r requirements/local.txt
    

      Maybe you will have some issues with permission in the folder ~/.cache, just perform the next command and problem solved:

           sudo chmod 777 ~/.cache

    if you deploy to Amazon AWS EC2 and you have a memory error try:

        pip install -r requirements/local.txt --no-cache-dir
    
  15. Start postgresql
  16.     sudo /etc/init.d/postgresql start
    

    Configure the database

  17. As root: (There should be a user postgres after installing the database)
  18.     su - postgres
    
  19. Create the database:
  20.     psql
    
        create user manati_db_user with password 'password';
    
        create database manati_db;
    
        grant all privileges on database manati_db to manati_db_user;
    
        alter role manati_db_user createrole createdb;
        
        CTRL-D (to output the postgres db shell)
    

    OPTIONAL

    To change the password by default of the postgres user (you can put the same password if you want), specially good idea if you want to use pgAdmin3-4 as a postgres client. Remember don't exit of "sudo - postgres"

        psql
        
        \password;
    
        CTRL-D (to output the postgres db shell)
    

    Verify that the db was created successfully

  21. As the postgres user
  22.     psql -h localhost -d manati_db -U manati_db_user
    
        (and put the password)
    

    After putting the password you should be logged in in the postgres.

  23. Install redis-server
  24.     sudo apt-get install redis-server
    

    OPTIONAL

    If you want to configure the Redis. For example, you are interested to change the password, you can:

        sudo vi /etc/redis/redis.conf
    

    and find the line requirepass and write next it the password that you want.

        requirepass passwodUser
    

    Just remember to update the variable environment REDIS_PASSWORD in the file .env in the root of the project.

  25. Run migrate files
  26.     python ./manage.py makemigrations guardian
        python ./manage.py migrate
    
  27. Registering External modules. You must run this command everytime you add or remove an External Module
  28.      python ./manage.py check_external_modules
    
  29. Execute redis_worker.sh file (in background '&' or in another console).
  30.     ./utility/redis_worker.sh
    
  31. Create super user for login in the web system if you need
  32.     python manage.py createsuperuser
    

    How to run it

    It is not recommended to run the server as root, but since only root can open ports numbers less than 1024, it is up to you which user you use. By default it opens the port 8000, so you can run it as root:

    python ./manage.py runserver
    

    After this, just open your browser in http://localhost:8000/manati_project/manati_ui

    If you want to open the server in the network, you can do it with:

    python ./manage.py runserver <ip-address>:8000
    

If you want to see the jobs running or enqueued go to http://localhost:8000/manati_project/django-rq/

Settings: Updating version from master

  1. Open project directory
  2. cd path/to/project_directory
    
  3. Pull the last changes from master
  4. git pull origin master
    
  5. Install las libraries installed
  6. pip install -r requirements/local.txt
    
  7. Install redis-server and execute redis_worker.sh file (in background '&' or in another console)
  8. ./utility/redis_worker.sh
    
  9. Prepare migrations files for guardian library (if it already has, nothings happens)
  10. python ./manage.py makemigrations guardian --noinput
    
  11. Execute migrations files
  12. python ./manage.py migrate --noinput
    
  13. Registering External modules. You must run this command everytime you add or remove an External Module
  14. python ./manage.py check_external_modules
    
  15. Execute server
  16. python ./manage.py runserver
    

Run in production.

Using surpevisor, gunicorn as server with RQ worker (with redis server) to deal with the background tasks. In the future we are planning to prepare settings for nginx

cd path/to/project_directory 
python manage.py collectstatic --noinput --clear
sudo supervisord -c supervisor-manati.conf -n

Docker image

If you have docker installed, maybe can be a good idea install the ManaTI docker image. The Dockerfile and server configurations files are here. This ManaTI docker image is executed over a server NGINX and uWSGI. This image is maintained by @Piuliss

docker pull honeyjack/manati:latest
docker run --name manati -p 8888:8888 -dti honeyjack/manati:latest bash

Then, wait for 5 or 10 seconds and go to http://localhost:8888

Docker Composer

If you don't want to waste time installing ManaTI and you have docker installed, you can just execute docker-compose. First clone the repository and go to the directory project.

cd Manati
cp .env.example .env
cp .env-docker.example .env-docker
docker-compose build
docker-compose run web bash -c "python manage.py makemigrations --noinput; python manage.py migrate; python manage.py check_external_modules"
docker-compose run web bash -c "python manage.py createsuperuser2 --username admin --password Password123 --noinput --email 'admin@manatiproject.com'"
docker-compose up # or 'docker-compose up -d' if you don't want to see the logs in the console.

After this, just open your browser in http://localhost:8000/manati_project/manati_ui/new

Backup DB

pg_dump -U manati_db_user -W -F p manati_db > backup.sql # plain text

Restore DB

psql manati_db -f backup.sql -U manati_db_user

Browser supported

IE / Edge
IE / Edge
Firefox
Firefox
Chrome
Chrome
Safari
Safari
Opera
Opera
Vivaldi
Vivaldi
N/A 55+ 60+ N/A N/A N/A

License

The GPLv3 License (GPLv3). See docs/LICENSE file for more details.

Copyright (c) 2016-2018 Stratosphere Laboratory