Intercepts data, does something with it, stores it - a part of the Logro project
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
app
scripts
.gitignore
.travis.yml
LICENSE
README.md
appspec.yml
requirements.txt

README.md

hungry-interceptor

This project intercepts/manipulates/requests/handles data. Then, it does one of many things:

  • Formats the data
  • Stores the data in a neo4j database.
  • Stores the data in a mongo database.
  • Stores the data in a different datastore entirely.
  • Kicks off an ETL process (planned Mid Oct 2016)
  • Etc.

This app is planned to be a gate keeper or router-like microservice for the rest of the engines, such as a machine learning engine (Non-existent in 2016), NLP engine ('speedy-affect-scorer' in 2016), ETL engine ('energetic-etl') or another engine.

Overview

Its primary function is getting data from its counterpart in the front end side of the Logro application (currently callled 'fixed-gateway' in 2016). Its other important function is sending data to its counterpart engines listed above via its mechanisms and API.

History

The project originally was designed to also do ETL functions which are now part of 'energetic-etl' (GPLv3) where the intercepts directory contained some of the logic this project used as boilerplate. See the other project called: energetic-etl

Find the project here: https://github.com/nbuechler/hungry-interceptor

Getting Started

  
    $ virtualenv venv
  
  • Next, activate the virtual environment (make sure you get the'.'):
  
    $ . venv/bin/activate
  
  • Last, install the requirements with pip:
  
    $ pip install -r requirements.txt
  

Most exciting future plans

I've begun working with a friend of mine in early 2016, and have found the usefulness in planning to intercept the data and send it to a machine learning model to produce similar results. This machine learning model is in a separate project and will likely be linked here when the project is in a stable state.

Technical challenges to overcome

  • Scaling the application to work in a cluster will be challenging due to the single-threaded nature of Flask.
  • Considering a separate microservice specifically for coordinating message queues, akin to something like Celery or RabbitMQ would be useful (but, I'm not sure yet if this is viable or a good idea)
  • It also might be valuable to rework this project to be more of a microservice, instead of a service-oriented project. Thus, looking into microservice architectures to apply to the entire Logro project is a good idea to consider.

Start databases - if they are not already running

From a terminal, start mongo:

  
    mongod
  

From a terminal start Neo4j:

  
    sudo /etc/init.d/neo4j-service start
  

Run the application

  
    python app/runserver.py 5000
  

This contains a port - 5000 - for running local, otherwise it will try to run on the default port - 80 - and that's taken. If you run it on a server instance, such as one on AWS without the port specified, it should running open to the world. That is the usual configuration for deploying code and making it 'live' to the world. But, make sure you are prudent in running the code in the way you want it to run.

Scripts folder

The scripts folder is for automation. If using AWS, here's a problem I encountered in the middle of Feb, 2016: DO NOT UPGRADE PIP from 6.1.1 on a standard AWS EC2 instance, or it will bite you. I don't really know why so this might have changed when you are reading this.

Requirements

  • Flask==0.10.1
  • Flask-Cors==2.1.0
  • Flask-PyMongo==0.3.1
  • Flask-WTF==0.11
  • itsdangerous==0.24
  • Jinja2==2.8
  • MarkupSafe==0.23
  • py2neo==2.0.8
  • pymongo==2.8.1
  • six==1.10.0
  • Werkzeug==0.11.3
  • wheel==0.24.0
  • WTForms==2.0.2

Things to do

  • Refactor: Remove old code from 'hungry-interceptor' when this is complete!

CORS and dealing with it

Make sure to pay attention to how CORS right now accepts everything.

See more here: https://flask-cors.readthedocs.org/en/latest/

  
    from flask.ext.cors import CORS
    cors = CORS(app, resources={r"/\*": {"origins": "\*"}}) #CORS :WARNING everything!
  

LICENSE

GPLv3