Threat Miner for SDL

A security tool from threat intelligence that uses web scraping and machine learning to identify potential threats in your products or components.

Prerequisites

1. A Python 2.7 and Pip installation
2. A Git installation
3. Install Node JS and NPM
4. install bower: npm i -g bower
5. A running MySQL web server. To run a MySQL web server locally, install WAMP if running on Windows, or install XAMPP if running on MAC/Linux. Create a database called 'threat_intelligence_db'
6. StanfordCoreNLP Downloaded with all jar files in the directory added to the "classpath" environment variable.

Workflow:

1. Installation Step
2. Usage Step
3. Make necessary code changes
4. Deployment Step

Installation

    git clone https://github.com/intel/threatminer-for-sdl
    cd threatminer-for-sdl
    pip install -r requirements.txt
    cd client
    npm install && bower install

Setup

    python setup.py

A prompt will present itself. Enter the following values:

    Host: localhost
    Username: root
    Password:
    DB: threat_intelligence_db
    Charset: utf8
    port: 3306
    caCert:
    isDeveloping: true

To create tables in the database (WARNING! DON'T RUN THIS ON AN EXISTING THREATMINER INSTANCE. IT MAY OVERWRITE YOUR TABLES)

    cd database
    python createDatabaseTables.py

Usage

If running windows. Note: you can store FLASK_APP and FLASK_ENV values as environment variables to avoid entering every time

    cd server
    set FLASK_APP=run.py
    set FLASK_ENV=development

If running Linux

    cd server
    export FLASK_APP=run.py
    export FLASK_ENV=development

Run the RestAPI:

    cd server
    flask run

Open a new terminal and run the client server:

    cd client
    gulp serve

Deployment

Clear the "isDeveloping" variable from setup.py

    python setup.py

If you are using https:

1. Uncomment 'if (window.location.protocol == "http:") {' block in client/app.js
2. Add cert file to server/, scripts/, and database/ directories
3. Run "setup.py" and enter the name of the cert file

Create the training models:

    cd scripts
    python train.py

Set serializeTo in scripts/classifier/configuration.prop to "<scripts/classifier's absolute path> + /ner-model.ser.gz"

Follow "Testing and deployment" step in docs/client for deploying front end

Follow "Deployment" step in docs/server for deploying Rest API

Follow "Virtual Mchine" step in docs/scripts for scheduling scripts