Skip to content

Documentation

Jump to bottom
Lenka Stejskalová edited this page Oct 25, 2018 · 6 revisions

DOCUMENTATION

SUPERVISOR

Configuration files are placed in /etc/supervisor.d/ directory as files *.ini. Every configuration file contains:

  • name of process,
  • command to run,
  • home directory,
  • attributes setting after-start trigger and count of attempts,
  • user,
  • log files.

Example of configuration file:

[program:supervisor_grip_celery]
command=celery -A tasks.app -Q celery worker --loglevel=INFO
directory=/home/grip
autostart=true
autorestart=true
startretries=3
user=root
stderr_logfile=/var/log/grip/celery.err.log
stdout_logfile=/var/log/grip/celery.out.log

RABBITMQ

Exchange celery and queue celery was created. Request will be send into this queue. Body of request is especially defined to recongized type of request. Queue is designed to process multiple types of requests. Every request message contains:

  • type of message,
  • identificator of request in UUID,
  • body od message (IDEA message),
  • optional arguments.

Request can be send by this code:

credentials = pika.PlainCredentials(’guest’, ’guest’)
parameters = pika.ConnectionParameters(’localhost’,
                                       5672,
                                       ’/’,
                                       credentials)
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
channel.basic_publish(exchange=’celery’,
                      routing_key=’celery’,
                      body=’{"task": "add",
                             "id": "’ + str(uuid.uuid4()) + ’",
                             "args": [’ + json.dumps(idea) + ’],
                             "kwargs": {}}’,
                      properties=pika.BasicProperties
                                 (content_type=’application/json’))
connection.close()

Every message is tag by content_type='application/json' to be readable as JSON.

CELERY

Process of Celery application are started with Supervisor with command

celery worker -A tasks --loglevel=INFO

which run 1 main process of Celery and 8 worker processes. Main process divide tasks into non-working workers, then worker process task according to rules in tasks.py. Celery use multiprocessing, so it can put task to only 1 worker.

Also Celery need some message broker from which it can recieve tasks. In this solution we use RabbitMQ message broker.

In file tasks.py there are defined functions for processing incomming messages - tasks. These functions are tag as @app.task, it means there are part of Celery solution. Whole tag looks like this:

@app.task(serializer=’json’, name=’add’)

Function takes message in JSON format. Every function can recieve only message with type tag same as tag of function, so function knowns how to process the message.File tasks.py imports file celeries.py, which contains definition of Celery application - name of application (just tag) and connection to message broker instance with connection to message broker instance for returning the results.

Celery application is run by command:

app = Celery(’celery’,
             backend=’amqp://localhost:5672’,
             broker=’amqp://localhost:5672’)
app.start()

POSTGRESQL

PostgreSQL database was install just for internal purpose. It holds data from incoming messages and searching is provided in these tables.

In database there are 4 tables: actual, history, uniq_ip and features.

NEO4J

Neo4j uses own query language Cypher inspired by SQL and with visual syntax of „ASCII art“. Example of query for creating node Ip with name 10.0.1.123 and node Note with atrribute id with value abcd and connectiong these two nodes with relationship DESCRIBES, returning node Ip:

CREATE (ip:Ip {name: 10.0.1.123})
CREATE (note:Note {id: abcd})
CREATE (note)-[:DESCRIBES]->(ip)
RETURN ip

Next example show the query searching for all nodes Group, which are in relationship PARTS_OF with some other node, which is in relationship DESCRIBES with node Note with id efgh, returns name and attribute reason of found node Group:

MATCH (note:Note {id: efgh})
MATCH (note)-[:DESCRIBES]->(ip)
MATCH (ip)-[:PARTS_OF]->(group)
RETURN group.name, group.reason ORDER BY group.name, group.reason

Neo4j was installed as server for CentOS system by the manual of creater. To control database from scripts we install Python module neo4j-driver. It needs to create new user as precaution from Neo4j, so we can't use default login.

PYTHON FLASK

Flask can process 3 types of request:

  • request for list of all found groups,
  • request for list of all found groups, which contain given IP address,
  • request for list of all found IP addresses, which are part of given ID of group.

Flask was installed as Python module. API is defined in file web.py, where all endpoints are defined and start of Flask application is defined. Application starts with attribute threaded=True, which means that multitreading is enabled. Three endpoints are defined, each for one type of request:

  • /grip/groups
  • /grip/groups/[IP_ADDRESS]
  • /grip/group/[ID_OF_GROUP]

Definition of endpoint contains route of endpoint, HTTP method and HTTP response returnig result. All endpoints are defined as GET methods. Result is returned by Flask function jsonify. Function return the result list of IP addresses or groups as Flask.Response() object, which contains content_type header 'application/json'. If the result is returning as json.dumps(), it would be coded string and MIME type header would miss.

Flask is running on Apache web server with mod_wsgi module. WSGI defined interface between web server and application.

There was created configuration file grip.cong for Apache web server, which sets access to IP addresses and domain names, defines parameters of allpication and set address of WSGI file. Also file wsgi.py was created, it defines application to run.

API

GRIP API is accessable only from localhost of GRIP. API is written in Flask web framework and deployed with WSGI module in Apache.

ROUTES

To find list of all found groups with reason for putting together:

/grip/groups/list

To find groups, which contain given IP address:

/grip/groups/<ip>

To find all IP addresses, which are part of given ID of group:

/grip/group/<id>

Response is in format of list of found IDs of groups and reason to create the group or list of IP addresses in group.

Request for list of all found groups

Definition of endpoint /grip/groups/:

@app.route(’/groups/’, methods=[’GET’])
def find_list():
    ...
    return jsonify(list)

Request:

curl https://localhost/grip/groups/

Result:

[
  [
    "7EXH1K8L5304RXRHO82AFN43ZU4PYOBO",
    "Ideal match"
  ],
  [
    "CT08FJVHHII1MA70B5O82T2D2TW18GXA",
    "Ideal match"
  ],
  [
    "VJCKVRQWH9EK9KTB9V7XKAAQ0N3PQ5FP",
    "Individual match on category"
  ],
...
  [
    "XL6125WHQBXNZLBRNA7R3CK1S6HMRFFM",
    "Ideal match"
  ]
]

Request for list of all found groups, which contain given IP address

Definition of endpoint /grip/groups/[IP_ADDRESS]:

@app.route(’/groups/<int:ip1>.<int:ip2>.<int:ip3>.<int:ip4>’, methods=[’GET’])
def find_ip(ip1, ip2, ip3, ip4):
    ...
    return jsonify(list)

Request:

curl https://localhost/grip/groups/10.1.11.12

Result:

[
  [
    "7EXH1K8L5304RXRHO82AFN43ZU4PYOBO",
    "Ideal match"
  ],
  [
    "CT08FJVHHII1MA70B5O82T2D2TW18GXA",
    "Ideal match"
  ]
]

Request for list of all found IP addresses, which are part of given ID of group

Definition of endpoint /grip/groups/[ID_OF_GROUP]:

@app.route(’/group/<string:botnet>’, methods=[’GET’])
def find_groups(botnet):
    ...
    return jsonify(list)

Request:

curl https://localhost/grip/group/CT08FJVHHII1MA70B5O82T2D2TW18GXA

Result:

[
  "150.129.133.100",
  "150.129.133.112",
  "150.129.133.13",
  "150.129.133.136",
  "150.129.133.140",
  "150.129.133.153",
  "150.129.133.176",
  "150.129.133.178",
  "150.129.133.192",
  "150.129.133.196",
  "150.129.133.208",
  "150.129.133.21",
  "150.129.133.221",
  "150.129.133.23",
  "150.129.133.37",
  "150.129.133.4",
  "150.129.133.46",
  "150.129.133.61",
  "150.129.133.74",
  "150.129.133.98"
]

Clone this wiki locally