Intercepts data, does something with it, stores it - a part of the Logro project
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.


This project intercepts/manipulates/requests/handles data. Then, it does one of many things:

  • Formats the data
  • Stores the data in a neo4j database.
  • Stores the data in a mongo database.
  • Stores the data in a different datastore entirely.
  • Kicks off an ETL process (planned Mid Oct 2016)
  • Etc.

This app is planned to be a gate keeper or router-like microservice for the rest of the engines, such as a machine learning engine (Non-existent in 2016), NLP engine ('speedy-affect-scorer' in 2016), ETL engine ('energetic-etl') or another engine.


Its primary function is getting data from its counterpart in the front end side of the Logro application (currently callled 'fixed-gateway' in 2016). Its other important function is sending data to its counterpart engines listed above via its mechanisms and API.


The project originally was designed to also do ETL functions which are now part of 'energetic-etl' (GPLv3) where the intercepts directory contained some of the logic this project used as boilerplate. See the other project called: energetic-etl

Find the project here:

Getting Started

    $ virtualenv venv
  • Next, activate the virtual environment (make sure you get the'.'):
    $ . venv/bin/activate
  • Last, install the requirements with pip:
    $ pip install -r requirements.txt

Most exciting future plans

I've begun working with a friend of mine in early 2016, and have found the usefulness in planning to intercept the data and send it to a machine learning model to produce similar results. This machine learning model is in a separate project and will likely be linked here when the project is in a stable state.

Technical challenges to overcome

  • Scaling the application to work in a cluster will be challenging due to the single-threaded nature of Flask.
  • Considering a separate microservice specifically for coordinating message queues, akin to something like Celery or RabbitMQ would be useful (but, I'm not sure yet if this is viable or a good idea)
  • It also might be valuable to rework this project to be more of a microservice, instead of a service-oriented project. Thus, looking into microservice architectures to apply to the entire Logro project is a good idea to consider.

Start databases - if they are not already running

From a terminal, start mongo:


From a terminal start Neo4j:

    sudo /etc/init.d/neo4j-service start

Run the application

    python app/ 5000

This contains a port - 5000 - for running local, otherwise it will try to run on the default port - 80 - and that's taken. If you run it on a server instance, such as one on AWS without the port specified, it should running open to the world. That is the usual configuration for deploying code and making it 'live' to the world. But, make sure you are prudent in running the code in the way you want it to run.

Scripts folder

The scripts folder is for automation. If using AWS, here's a problem I encountered in the middle of Feb, 2016: DO NOT UPGRADE PIP from 6.1.1 on a standard AWS EC2 instance, or it will bite you. I don't really know why so this might have changed when you are reading this.


  • Flask==0.10.1
  • Flask-Cors==2.1.0
  • Flask-PyMongo==0.3.1
  • Flask-WTF==0.11
  • itsdangerous==0.24
  • Jinja2==2.8
  • MarkupSafe==0.23
  • py2neo==2.0.8
  • pymongo==2.8.1
  • six==1.10.0
  • Werkzeug==0.11.3
  • wheel==0.24.0
  • WTForms==2.0.2

Things to do

  • Refactor: Remove old code from 'hungry-interceptor' when this is complete!

CORS and dealing with it

Make sure to pay attention to how CORS right now accepts everything.

See more here:

    from flask.ext.cors import CORS
    cors = CORS(app, resources={r"/\*": {"origins": "\*"}}) #CORS :WARNING everything!