Find file
Fetching contributors…
Cannot retrieve contributors at this time
309 lines (231 sloc) 12.4 KB
0) Clone a copy of the skeleton:
git clone
0a) I use tcsh(1) for day-to-day use and then sh(1) for scripting. Most
bash(1) enthusiasts don't have enough perspective to have an opinion
worth listening to. New tcsh users, enjoy the up-arrow history,
tab-completion for ssh hosts, `time long_command`, etc., ...
See for details, or just:
$ curl -o ~/.cshrc
$ less ~/.cshrc # _Always_ review changes to your shell config
$ exec /bin/tcsh # Start using tcsh(1)
% chsh /bin/tcsh # Change your default shell once you feel comfortable
% mv .enter.tcsh.dist .enter.tcsh; mv .enter.local.tcsh.dist .enter.local.tcsh
% ${EDITOR} .enter.tcsh .enter.local.tcsh
% cd $PWD # Run this after you finish installing PostgreSQL
0b) Download and install python 2.7.1 (NOTE: python 3.X is a no-go with
Flask - don't get clever)
% sudo port install python27
0c) Download and install libmemcached >= 0.4.0. I'm going to leave this as a
an exercise to the reader as to how to best do this in your environment,
but if you're on a mac with MacPorts:
% sudo port install libmemcached
0d) Download and install PostgreSQL. Similar to the above, but since this is
a database and not a random library, you're on your own for not being a
% sudo port install postgresql91-server
% sudo mkdir -p /opt/local/var/db/postgresql91/defaultdb
% sudo chown postgres:postgres /opt/local/var/db/postgresql91/defaultdb
# Run this in a different Terminal window
% sudo su postgres -c '/opt/local/lib/postgresql91/bin/initdb -D /opt/local/var/db/postgresql91/defaultdb'
# Not required, but man pages are handy!
% sudo port install postgresql91-doc
1) Rename from 'skeleton' to 'myapp':
# At some point I'll change this step to be scripted (hell, I might even
# actually test this step at some point)
find . -type f -print0 | xargs -0 -n 1 perl -p -i -e '#skeleton#myapp#go'
mv skeleton myapp
# Search for and replace the remaining references to 'skeleton'
egrep -ri skeleton *
find . | grep -i skeleton
2) Start the database:
# This should be set by your .enter.local.tcsh file.
#setenv PGUSER_ROOT pgsql
# MacPorts installed version of PostgreSQL 9.1, I run this in a different
# terminal when developing so I can see what's going on.
sudo su - ${PGUSER_ROOT} -c '/opt/local/lib/postgresql91/bin/postgres -D /opt/local/var/db/postgresql91/defaultdb'
3) Create a virualenv for the skeleton:
# MacPorts installed version of Python 2.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/virtualenv --no-site-packages .
4) Activate the virtualenv:
# If you installed the .cshrc file mentioned above and use tcsh(1), just do this:
chmod 0600 .enter.tcsh
cd $PWD
# To manually activate the virtualenv, run the following:
source bin/activate.csh
5) Pull in the required packages.
# Tweak the build environment slightly for python-libmemcached and
# M2Crypto.
setenv CPPFLAGS '-I/opt/local/include -I/opt/local/include/openssl'
# Break the installation down in to two steps: download and install (if
# something fails, you don't want to start over again from scratch). ~/tmp
# is created by .cshrc.
mkdir ~/tmp/pip
pip install -U --download-cache=~/tmp/pip -I -r requirements.txt --no-install
pip install -U --download-cache=~/tmp/pip -I -r requirements.txt --no-download
# Make sure everything is there (or newer):
pip freeze | sort > requirements.txt
git diff requirements.txt
5.1) Patch(es).
# For now, Werkzeug doesn't support python-libmemcached, but it can by
# applying the following patch to your install (versions 0.6.2 & 0.7.0):
% patch -d lib/python2.7/site-packages/ -p0 < patches/werkzeug::contrib::cache_452e8402d056.patch-0.6.2
6) Setup debugging:
echo 'DEBUG = True' >>
echo 'TESTING = False' >>
When you go in to production, be sure to remove these two lines!!!
7) Initialize the database (these steps should be moved in to a script for
easy unit testing):
# l2dba 101.
# Environment variables that should be set by your .enter.local.tcsh:
setenv PGUSER_DBA skeleton_root
setenv PGUSER_ROOT pgsql
# Create the base roles
psql -d template1 -U ${PGUSER_ROOT} -1f sql/initialize/100_create_roles.sql
# Create your user roles (use two, one for read-only and the other for
# read-write activities)
# Add the ROLEs to their respective DBA ROLE/GROUPs
psql -d template1 -U ${PGUSER_ROOT} -c "ALTER GROUP skeleton_dba ADD USER ${PGUSER};"
psql -d template1 -U ${PGUSER_ROOT} -c "ALTER GROUP skeleton_root ADD USER ${PGUSER_RW};"
# Create the rest of the DB objects
psql -d template1 -U ${PGUSER_ROOT} -f sql/initialize/130_create_database.sql
psql -U ${PGUSER_ROOT} -1f sql/initialize/140_alter_public_schema.sql
# Load the pgcrypto and uuid functions in to skeleton
cat /opt/local/share/postgresql91/extension/pgcrypto--1.0.sql | sed -e 's#MODULE_PATHNAME#pgcrypto#' > ~/tmp/pgcrypto.sql
cat /opt/local/share/postgresql91/extension/uuid-ossp--1.0.sql | sed -e 's#MODULE_PATHNAME#uuid-ossp#' > ~/tmp/uuid-ossp.sql
psql -U ${PGUSER_ROOT} -1f ~/tmp/pgcrypto.sql
psql -U ${PGUSER_ROOT} -1f ~/tmp/uuid-ossp.sql
# Load the schema
psql -U ${PGUSER_DBA} -1f sql/initialize/200_schema.sql
psql -U ${PGUSER_DBA} -1f sql/initialize/205_integrated_definitions.sql
psql -U ${PGUSER_DBA} -1f sql/initialize/210_tables.sql
psql -U ${PGUSER_DBA} -1f sql/initialize/280_views.sql
# Load the functions. This loads the functions and changes their
# permissions. Change DATABASE_PASSWORD_HASH to a random string of bytes
# that's at least 32 bytes in length. Please use secure key handling
# techniques and make sure that the DATABASE_PASSWORD_HASH key only exists
# on your database servers and is not present (in any capacity) along side
# with your webserver password keys.
pwgen -acn 32 1 > sql/db_password.hash
psql -At -U ${PGUSER_DBA} -c 'SELECT uuid_generate_v4();' > sql/db_email.uuid
chmod 600 sql/db_password.hash sql/db_email.uuid
sed -e "s#DATABASE_PASSWORD_HASH#`cat sql/db_password.hash`#" \
-e "s#DATABASE_EMAIL_UUID#`cat sql/db_email.uuid`#" \
sql/initialize/ > sql/initialize/300_funcs.sql
psql -U ${PGUSER_DBA} -1f sql/initialize/300_funcs.sql
# CREATE TRIGGER commands are run outside of funcs.sql because there is no
psql -U ${PGUSER_DBA} -1f sql/initialize/400_triggers.sql
# ALTER FUNCTION ... OWNER TO ... commands must be run by the -U ${PGUSER_ROOT} user.
psql -U ${PGUSER_ROOT} -1f sql/initialize/500_fixup_func_owner.sql
# Add aaa and mod1 to the skeleton_www user's default search_path. This
# only needs to be done when you create a new user.
psql -U ${PGUSER_ROOT} -c "ALTER ROLE skeleton_www IN DATABASE ${PGDATABASE} SET search_path = aaa, mod1, public"
# Populate some initial data
psql -U ${PGUSER_DBA} -1f sql/initialize/800_initial_data.sql
psql -U ${PGUSER_DBA} -1f sql/initialize/801_timezone_data.sql
# Setup the GRANTS. Be sure to audit the database permissions after the
# fact via the psql(1) commands: \dp, \ddp and \dn+
psql -U ${PGUSER_DBA} -1f sql/maintenance/100_perms.sql
# Neuter the DBA user and use your own personal DBA account.
psql -U ${PGUSER_ROOT} -1f sql/initialize/900_cleanup_users.sql
# Other misc PostgreSQL tips:
# 1) Look in to using the contrib'ed auto-explain module:
# 2) A few suggested development tunables for postgresql.conf:
# shared_preload_libraries = 'pgcrypto,uuid-ossp'
# synchronous_commit = off
# log_connections = on
# log_disconnections = on
# log_duration = on
# log_statement = 'all'
# timezone = 'UTC'
# 3) Learn how to use dblink (and get creative with its use in functions
# and views):
# 4) Always use pgbouncer ( I
# recommend using PostgreSQL on and then pgbouncer on
# *:5432. Be sure to check out the 'pgbouncer' database and run the
# 'SHOW STATS' commands.
# 5) Small reading comprehension test: until you set the database or the
# connection's timezone to UTC, timezone to UTC, you will get
# '*check1'-like errors.
8) Run the app:
# The first few times you run, it will puke up errors and
# have you add various bits to It is critically
# important that:
# a) PASSWORD_HASH: All webservers have the same PASSWORD_HASH. If you
# loose PASSWORD_HASH, users will not be able to log in. The contents
# of PASSWORD_HASH should never exist on your database servers.
# b) sql/db_password.hash: The contents of sql/db_password.hash should
# never be present on your webservers. Like PASSWORD_HASH, if you
# loose the contents of db_password.hash, users will not be able to
# log in without resetting their password (on the plus side, at least
# db_password.hash is stored in the contents of the pl functions -
# which are visible to any logged in user, regardless of the
# permissions set on the function's enclosing schema and the function
# itself).
# c) SECRET_KEY: SECRET_KEY must be synchronized across all webservers,
# too, but unlike PASSWORD_HASH, you can regenerate it at any time and
# there's no real loss (users will have to re-login).
# d) BROWSER_SECRET_KEY: Similar story to SECRET_KEY. All web servers
# must have the same BROWSER_SECRET_KEY. If you loose this key, every
# BrowserID cookie will be reset. Not the end of the world, but
# tedious as hell to recover from.
# certs. Don't loose them. Don't use Werkzeug for handling SSL in
# production, but it's really convenient to test in an SSL environment
# from the beginning.
a) Register:
Then update shadow.aaa_email_confirmation_log:
skeleton=> UPDATE shadow.aaa_email_confirmation_log SET confirmed = TRUE WHERE id = 1;
9) In production, you will need to combine all of your static assets
together and probably have them served by nginx. Something like:
mkdir static
cd static
find ../skeleton -name static
# Make a symlink for everything in to the static dir. e.g.:
# ln -s ../skeleton/modules/mod1/static mod1
# Be sure to change your CANONICAL_NAME and CANONICAL_PORT settings once
# you move in to production.
10) Feel free to re-run the sql/maintenance/100_perms.sql script as many times as you'd
like. \dp, \ddp and \dn+ are your friends.
11) Debugging notes. With your environment activated:
# Nearly every time I need to do any first time work with SQLAlchemy (or
# any real database work), I fire up a terminal window and execute the
# following commands:
[flask-skeleton] % python
>>> from skeleton import create_app
>>> app = create_app()
>>> ctx = app.test_request_context()
>>> ctx.push()
>>> from skeleton import db
>>> ses = db.session
# Which is why I dump all of that stuff in a file:
% python -i
# From here, I can do useful things and figure out how to map objects
# together in the ORM without pulling my hair out.
>>> from mod3.models import Tag, Page
>>> p = Page.query.filter_by(id = 1).first()
>>> p.__dict__
{'url': u'', '_sa_instance_state': <sqlalchemy.orm.state.InstanceState object at 0x1023a9490>, 'id': 1}
>>> dir(p)
>>> t = Tag.query.filter_by(name = 'mytag').first()
>>> t.pages.filter( is None
# I frequently dump the tedium of that kind of debugging in to debugging
# scripts (pro tip: echo '/debug_*' >> .git/info/exclude)
>>> from debug_mod3 import *