Save job information to a database [DON'T MERGE]

[lukeyeager]

Motivation

• Allows communication between server and workers on different nodes (coming soon!)
• May enable multi-threaded webserver (gunicorn + nginx issues #479)
  • Need to investigate compatibility with Flask-SocketIO
• Faster startup/shutdown times
• Browse/edit job data with SQL tools

Features

• You can use SQLite, MySQL, PostgreSQL or anything else supported by SQLAlchemy
• Database schema changes are handled automatically through Flask-Migrate
• New CLI using Flask-Script

  • Start development server

    $ ./manage.py runserver --help
    usage: manage.py runserver [-h] [-t HOST] [-p PORT] [-d] [-r]
    
    Runs the Flask-SocketIO development server
    
    optional arguments:
    -h, --help            show this help message and exit
    -t HOST, --host HOST
    -p PORT, --port PORT
    -d, --no-debug
    -r, --reload

  • Create / upgrade the database

    ./manage.py db init
    ./manage.py db migrate
    ./manage.py db upgrade

Progress

• Define db schema
• Load data into db from previous jobs (pickle files)
• Store data for new jobs to db
• Read job data from db
• Stop loading pickle files
• Stop saving pickle files
• Set up cascading deletion for appropriate relationships
• Add configuration option for database dialect
• Document how to create and connect to a database

[lukeyeager]

Saving to lukeyeager:backup/sql in case we revisit this.