django-kevin

A heavily personalized project template for Django 1.8.4 using Postgres for development and production. Ready to deploy on Heroku with a bunch of other goodies.

Forked from the original django-two-scoops-project

Creating Your Project

Prerequisites: Django

To create a new Django project, run the following command replacing PROJECT_NAME with your actual project name:

django-admin.py startproject --template=https://github.com/imkevinxu/django-kevin/archive/master.zip --extension=py,md,html,json,coveragerc PROJECT_NAME

Afterwards please reference the actual README.md you just created in your new project folder, all the references to "{{ project_name }}" will be changed accordingly.

Make virtual environments

Prerequisites: virtualenv, virtualenvwrapper

cd {{ project_name }}
mkvirtualenv {{ project_name }}-dev && add2virtualenv `pwd`
mkvirtualenv {{ project_name }}-prod && add2virtualenv `pwd`
mkvirtualenv {{ project_name }}-test && add2virtualenv `pwd`

Install python packages

For development:

workon {{ project_name }}-dev
sudo pip install --upgrade pip
sudo pip install --upgrade setuptools
sudo env ARCHFLAGS="-arch i386 -arch x86_64" pip install psycopg2
sudo pip install -r requirements/dev.txt

For production:

workon {{ project_name }}-prod
sudo pip install --upgrade pip
sudo pip install --upgrade setuptools
sudo env ARCHFLAGS="-arch i386 -arch x86_64" pip install psycopg2
sudo pip install -r requirements.txt

For testing:

workon {{ project_name }}-test
sudo pip install --upgrade pip
sudo pip install --upgrade setuptools
sudo env ARCHFLAGS="-arch i386 -arch x86_64" pip install psycopg2
sudo pip install -r requirements/test.txt

Install node packages

Prerequisites: node

sudo npm install

Sometimes the install may stall or not install everything. Try running npm list and then manually installing anything that may be missing.

One-time system installs

Prerequisites: homebrew

In order to use the grunt task runner you need to install it globally:

sudo npm install -g grunt-cli

In order to be able to lint SCSS files locally you need ruby on your local system and a certain gem. See https://github.com/ahmednuaman/grunt-scss-lint#scss-lint-task

gem install scss-lint

In order to use django-pipeline for post-processing, you need yuglify installed on your local system:

sudo npm install -g yuglify

In order for grunt to notify you of warnings and when the build is finished, you need a notification system installed. Below is the Mac OSX notification command-line tool:

brew install terminal-notifier

In order to use Redis for caching and queuing, you need to download it and have it running in the background. This will also set redis-server to automatically run at launch:

brew install redis
ln -sfv /usr/local/opt/redis/*.plist ~/Library/LaunchAgents
launchctl load ~/Library/LaunchAgents/homebrew.mxcl.redis.plist
launchctl start ~/Library/LaunchAgents/homebrew.mxcl.redis.plist

Development Mode

Set .env.dev variable for dev

The environment variables for development sets the appropriate DJANGO_SETTINGS_MODULE and PYTHONPATH in order to use django-admin.py seemlessly. Necessary for Foreman and other worker processes

.env.dev is not version controlled so the first person to create this project needs to create a .env.dev file for Foreman to read into the environment. Future collaboraters need to email the creator for it.

echo DJANGO_SETTINGS_MODULE=config.settings.dev >> .env.dev
echo PYTHONPATH={{ project_name }} >> .env.dev
echo PYTHONUNBUFFERED=True >> .env.dev
echo PYTHONWARNINGS=ignore:RemovedInDjango19Warning >> .env.dev
echo CACHE=dummy >> .env.dev

Recommended to use foreman to use development environment variables and processes:

echo "env: .env.dev" > .foreman
echo "procfile: Procfile.dev" >> .foreman

Compile initial static assets

This will compile all the files in /{{ project_name }}/static for the first run.

grunt build

Create local postgres database for dev

Prerequisites: Postgres and Heroku Toolbelt

Install Postgres for your OS here. For Max OSX the easiest option is to download and run Postgres.app.

# Make sure Postgres.app is running
workon {{ project_name }}-dev
createdb {{ project_name }}-dev
foreman run django-admin.py migrate

Run project locally in dev environment

Use the right virtual environment:

workon {{ project_name }}-dev

Start the server with:

foreman start

Create a local super user with:

foreman run django-admin.py createsuperuser

To run one-off commands use:

foreman run django-admin.py COMMAND

To enable Live Reload, download and turn on a browser extension.

Production Mode

Set .env variable for prod

The environment variables for production must contain a separate SECRET_KEY for security and the appropriate DJANGO_SETTINGS_MODULE and PYTHONPATH in order to use django-admin.py seemlessly. Hacky use of date | md5 to generate a pseudo-random string.

.env is not version controlled so the first person to create this project needs to create a .env file for Foreman and Heroku to read into the environment. Future collaboraters need to email the creator for it.

echo -n SECRET_KEY=`date | md5` >> .env
sleep 1
echo `date | md5` >> .env
echo DJANGO_SETTINGS_MODULE=config.settings.prod >> .env
echo PYTHONPATH={{ project_name }} >> .env
echo WEB_CONCURRENCY=3 >> .env
echo PYTHONUNBUFFERED=True >> .env
echo PYTHONWARNINGS=ignore:RemovedInDjango19Warning >> .env
echo BUILDPACK_URL=https://github.com/heroku/heroku-buildpack-multi.git >> .env

Deploy to Heroku

Prerequisites: Heroku Toolbelt and heroku-config

First step is to deploy to Heroku with the post_compile script in /bin so that node functions can be installed for python to call them.

git init
git add .
git commit -m "Ready for initial Heroku deploy"
heroku create
heroku config:push
git push heroku master

After post_compile is successful, uncomment the line with the variable STATICFILES_STORAGE in /{{ project_name }}/config/settings/base.py to enable django-pipeline and push again.

git commit -am "Enabled django-pipeline"
git push heroku master
heroku run django-admin.py migrate
heroku open

To run one-off commands like createsuperuser use:

heroku run django-admin.py COMMAND

Debugging tip: sometimes purging the cache can help fix a random error. Just run:

heroku run django-admin.py clear_cache

Run project locally in prod environment

Set the .foreman file to use production environment variables and processes:

echo "env: .env" > .foreman
echo "procfile: Procfile" >> .foreman

Use the right virtual environment:

workon {{ project_name }}-prod

This is meant to mimic production as close as possible using both the production database and environment settings so proceed with caution.

WARNING: If this project has SSL turned on, localhost:5000 won't work anymore because it will always try to redirect to https://localhost:5000. To fix this comment out the SECURITY CONFIGURATION section in /{{ project_name }}/config/settings/prod.py

heroku config:pull
foreman run django-admin.py collectstatic --noinput
foreman start

The site will be located at localhost:5000

Testing Mode

Set .env.test variable for test

The environment variables for testing sets the appropriate DJANGO_SETTINGS_MODULE and PYTHONPATH in order to use django-admin.py seemlessly. Necessary for Foreman and other worker processes

.env.test is not version controlled so the first person to create this project needs to create a .env.test file for Foreman to read into the environment. Future collaboraters need to email the creator for it.

echo DJANGO_SETTINGS_MODULE=config.settings.test >> .env.test
echo PYTHONPATH={{ project_name }} >> .env.test
echo PYTHONUNBUFFERED=True >> .env.test
echo PYTHONWARNINGS=ignore:RemovedInDjango19Warning >> .env.test

Run tests locally in test environment

Set the .foreman file to use testing environment variables and processes:

echo "env: .env.test" > .foreman
echo "procfile: Procfile.test" >> .foreman

Use the right virtual environment:

workon {{ project_name }}-test

And have static assets prepared (for coverage tests):

grunt build
foreman run django-admin.py collectstatic --noinput

Automatically run all tests and linters and watch files to continuously run tests:

foreman start

You can view the results of the tests in HTML at localhost:9000/tests

You can specifically view the results of Django coverage tests at localhost:9000/tests/django

Jasmine JS Unit Tests

Grunt automatically compiles Jasmine tests written in CoffeeScript at /{{ project_name }}/static/js/tests/coffee and runs the tests upon every save.

You can specifically view the results of Jasmine JS unit tests at localhost:9000/tests/jasmine

You can specifically view the results of JS coverage tests at localhost:9000/tests/jasmine/coverage.html

Add-ons & Services

SSL

Enable SSL via Heroku, Cloudflare, or your DNS provider and then uncomment the SECURITY CONFIGURATION section in /{{ project_name }}/config/settings/prod.py to enable security best practices for production.

Invoke

Scripts can be programmed to be run on the command-line using Invoke for repeated tasks like deployment, building, or cleaning. Write your tasks in tasks.py.

Redis Cloud Caching

In order to enable redis for caching and queues, add Redis Cloud to Heroku.

heroku addons:add rediscloud:25

Redis Queue Worker

Add a Redis Queue worker process to Procfile:

echo "worker: django-admin.py rqworker high default low" >> Procfile

Push the changed Procfile to Heroku:

git add Procfile
git commit -m "Added worker process to Procfile, pushing to Heroku"
git push heroku master

Turn on background job worker with this one-liner:

heroku scale worker=1

Redis Queue Scheduler

Add a RQ Scheduler process to Procfile:

echo "scheduler: rqscheduler --url \$REDISCLOUD_URL" >> Procfile

Push the changed Procfile to Heroku:

git add Procfile
git commit -m "Added scheduler process to Procfile, pushing to Heroku"
git push heroku master

Turn on background job scheduler with this one-liner:

heroku scale scheduler=1

Amazon S3

To use Amazon S3 as a static and media file storage, create a custom Group and User via IAM and then a custom static bucket and media bucket with public read policies.

Add the following config variables to Heroku:

heroku config:set AWS_ACCESS_KEY_ID=INSERT_ACCESS_KEY_ID
heroku config:set AWS_SECRET_ACCESS_KEY=INSERT_SECRET_ACCESS_KEY
heroku config:set AWS_STATIC_STORAGE_BUCKET_NAME={{ project_name }}-static
heroku config:set AWS_MEDIA_STORAGE_BUCKET_NAME={{ project_name }}-media

Monitoring

Librato for Heroku performance monitoring
New Relic for server performance monitoring (protip: set availability monitoring on to avoid Heroku idling)
RedisMonitor for Redis server monitoring
Logentries provides logging backups as well as search and notifications. Can also additionally backup to S3
Sentry for error tracking with Raven as the client. Make sure to use a synchronous blocking transport.
Ranger to alert you when your app is down

Testing

Rainforest QA for simple integration testing
Tinfoil Security for regularly scanning your app for security vulnerabilities
Loader.io for load testing

Continuous Integration

Includes a fancy badge for GitHub README

Travis CI for continuous integration testing
Coveralls.io for coverage testing
Requires.io for dependency management

Utilities

Libraries

Python 2.7.9

Currently using Django 1.8.4 for the app framework

base.txt

bpython 0.14.2 - Advanced python interpreter/REPL
defusedxml 0.4.1 - Secure XML parser protected against XML bombs
dj-static 0.0.6 - Serve production static files with Django
django-authtools 1.2.0 - Custom User model classes such as AbstractEmailUser and AbstractNamedUser
django-braces 1.8.1 - Lots of custom mixins
django-clear-cache 0.3 - Simple Django management command that clears your cache
django-extensions 1.5.6 - Useful command line extensions (shell_plus, create_command, export_emails)
django-floppyforms 1.5.2 - Control of output of form rendering
django-model-utils 2.3.1 - Useful model mixins and utilities such as TimeStampedModel and Choices
django-pipeline 1.5.4 - CSS and JS compressor and compiler. Also minifies HTML
django-redis 4.2.0 - Enables redis caching
django-rq 0.8.0 - Django integration for RQ
invoke 0.10.1 - Python task execution in tasks.py
logutils 0.3.3 - Nifty handlers for the Python standard library’s logging package
project-runpy 0.3.1 - Helpers for Python projects like ReadableSqlFilter
psycopg2 2.6.1 - PostgreSQL adapter
python-magic 0.4.6 - Library to identify uploaded files' headers
pytz 2015.4 - World timezone definitions
requests 2.7.0 - HTTP request API
rq-scheduler 0.5.1 - Job scheduling capabilities to RQ
six 1.9.0 - Python 2 and 3 compatibility utilities
static 1.1.1 - Serves static and dynamic content
unicode-slugify 0.1.3 - A slugifier that works in unicode

dev.txt

Werkzeug 0.10.4 - WSGI utility library with powerful debugger
django-debug-toolbar 1.3.2 - Debug information in a toolbar
django-sslserver 0.15 - SSL localhost server

prod.txt

Collectfast 0.2.3 - Faster collectstatic
boto 2.38.0 - Python interface to AWS
dj-database-url 0.3.0 - Allows Django to use database URLs for Heroku
django-storages 1.1.8 - Custom storage backends; using S3
gunicorn 19.3.0 - Production WSGI server with workers

test.txt

coverage 3.7.1 - Measures code coverage
nose-exclude 0.4.1 - Easily specify directories to be excluded from testing
django-nose 1.4.1 - Django test runner using nose
factory-boy 2.5.2 - Test fixtures replacement for Python
flake8 2.4.1 - Python style checker

config/lib

colorstreamhandler.py - Colored stream handler for python logging framework
finders.py - Custom Django finders with ignore setting
tdaemon.py - Test daemon in Python modified to work with django-admin.py, django-nose, and coverage

Node 0.12.X

Currently using NPM engine 2.X. Purpose is to watch and compile frontend files

dependencies

browserify 11.0.1 - Browser-side require() the node.js way
coffee-script 1.10.0 - Cleaner JavaScript
grunt 0.4.5 - Automatic Task Runner
grunt-cli 0.1.13 - Grunt's command line interface
grunt-autoprefixer 3.0.3 - Parse CSS and add vendor-prefixed CSS properties
grunt-contrib-clean 0.6.0 - Clear files and folders
grunt-contrib-coffee 0.13.0 - Compile CoffeeScript files to JavaScript
grunt-contrib-imagemin 0.9.4 - Minify PNG, JPEG, GIF, and SVG images
grunt-contrib-jasmine 0.9.1 - Run jasmine specs headlessly through PhantomJS
grunt-sass 1.0.0 - Compile Sass to CSS
grunt-template-jasmine-istanbul 0.3.4 - Code coverage template mix-in for grunt-contrib-jasmine, using istanbul
grunt-text-replace 0.4.0 - General purpose text replacement for grunt
load-grunt-config 0.17.2 - Grunt plugin that lets you break up your Gruntfile config by task
time-grunt 1.2.1 - Display the elapsed execution time of grunt tasks
yuglify 0.1.4 - UglifyJS and cssmin compressor