Skip to content
Insights into Forks
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
app
.gitignore
.travis.yml
LICENSE
README.md
celery_worker.py
config.ini
config.py
environment.yaml
manage.py
requirements.txt
wsgi.py

README.md

INFOX Build Status Codacy Badge

Website: http://forks-insight.com

Another related repo: https://github.com/shuiblue/INFOX

Language: Python3

Framework: Flask

Database: mongo

Http server: uwsgi & nginx

More on Wiki Page

Quick Start:

  1. Ramp up the environment according to environment.yaml(or requirements.txt)

    Here is an example of using Anaconda:

  • install conda (python3 version) Download Anaconda

  • install dependencies using environment.yaml

    conda env create -f environment.yaml
    source activate p3  (p3 is the env's name, see in environment.yaml)
  1. Install mongodb & redis

  2. Edit the config (see in config.py) & Set the environment variables

    1. Check the config.py

    2.    export GITHUB_CLIENT_ID=[your_github_oAuth_Client_ID]
      
         export GITHUB_CLIENT_SECRET=[your_github_oAuth_Client_Secret]
      
         export INFOX_LOCAL_DATA_PATH=[local path for storing analyzed result (like /Users/fancycoder/infox_data)]
      
         export INFOX_SECRET_KEY=[a random string(like abcd1234)]
      
         export INFOX_MAIL_USERNAME=[smtp_username]
      
         export INFOX_MAIL_PASSWORD=[smtp_password]
  3. Run http server on localhost:

    python manage.py runserver --threaded
  4. Run worker for async crawling on localhost:

    celery worker -A celery_worker.celery --loglevel=info

    Use flower to monitor the worker:

    celery flower --port=5555 --broker=redis://localhost:6379/0 --broker_api=redis://localhost:6379/0  
  5. Deploy on server:

    An quick tutorial: A Simple Tutorial for deploying your Flask application with uWSGI + nginx on server without root permission

    Another online tutorial: Serve Flask Applications with uWSGI and Nginx on Ubuntu 16.04

Architecture Overview:

code_architecture

Main Part

./app/main - Program Entrance

./app/analyse - Crawler

./app/analyse/analyser.py - Start Crawler and do analysis, load result into database.

./app/analyse/compare_changes_crawler.py - comparing the diff bewteen two repos.

./app/analyse/clone_crawler.py - Download the source code for repo, prepare for calculation for keywords.

./models.py - Database Model

./app/auth - Logic about account

./app/templates - HTML files(related to ./app/main/views.py)

./app/static - CSS/Javascript/Img Resource

./app/tests - Basic Test

Configuration Files

./config.py - Config for Flask

./config.ini - Config for using uWSGI

./wsgi.py - Start script for uwsgi

./celery_worker.py - Start script for crawler worker

./manage.py - Start script for testing

./requirements.txt - lib install for pip install

./environment.yaml - env for anaconda

Crawler Part

Under ./app/analyse

./app/analyse/analyser.py is the entrance of crawler.

Following is the workflow.

workflow1

workflow2

You can’t perform that action at this time.