Increase transaction isolation level

[zwn]

From #617 (comment):

• We need to increase transaction isolation either to 'repetable read' or 'serializable' (possibly in the whole app).
• To be able to do that, all places need to be able to deal with serialization errors.
• To be able to do that, the parts of code we rerun need to be idempotent.

[chadwhitacre]

This is ★★★. Shouldn't it be DevX★?

[zwn]

Hard to tell. It can result in problems for the user that we might not be able to track back to a particular line of code since it will be similar to race condition. Anyway, relabeled.

[zwn]

Postgres docs to set the levels: http://www.postgresql.org/docs/9.3/static/sql-set-transaction.html

[zwn]

ALTER DATABASE can be used to set the default transaction isolation level. I am wondering if the best way forward is to set this at the database level and see what breaks and fix it afterwards. What can happen is that psycopg2.extensions.TransactionRollbackError is thrown when the serializable property is violated. When such a thing happens the solution is to try it again (which current code does not do). Converting all code in gittip to be able to retry before enabling this unnecessarily prolongs the time when serialization errors might destabilize the consistency of our data. Not knowing about it does not mean it is not happening. Enabling it for all transactions at least lets us know about the size of the problem.

@whit537 @bruceadams Opinions?

BTW: default SQL level is 'serializable'. Having 'read commited' as the default is a postgres thing.

[zwn]

I tried it locally and I get few of them, all in set_session_expires. Hmm, I remember this code, it tries to write to the database on each access (even for requests returning 304). I'll fix it in a separate PR.

TransactionRollbackError: could not serialize access due to concurrent update

[Changaco]

I don't think we want to do that, it's complicated and the gain is questionable.