Rewrite payday

[Changaco]

Opening an issue as discussed in IRC.

Here are the main goals of the rewrite:

• make it fast: having it run in under 10 minutes would be nice
• make it safe: we've had failures in the past, we need to do better, it's money we're handling here
• reduce the amounts of fees and escrow by not charging credit cards when we don't actually need to

And here's my checklist:

• solve credit card statement should only be for the difference #1486 by implementing card holds
• do stuff directly in SQL instead of making a lot of round trips between the DB and the python code:
  • update stats in a single request at the end of payday
  • rewrite the transfer of tips and takes in SQL
• optimize the SQL (explain analyze FTW)
• do payin in a single transaction
• fix take_over during payday running can result in transfer to a dead account #1853
• make sure we don't get out of sync with Balanced (tighten up payday script around exchanges #213)
• make sure we don't transfer tips to participants who don't want to receive them (modify payday to not process gifts to participants with goal < 0 #916)
• investigate take clipping is overzealous? #2537
• handle payout failures (Failed ACH transfers aren't reflected in user balance #1811)
• parallelize the Balanced API requests

(Am I missing other issues that the payday rewrite should fix ?)

[chadwhitacre]

do everything in a single transaction

Why? Not doing everything in a single db transaction was an intentional design decision to enable better parallelization and crash recovery.

[Changaco]

But payday is not actually parallelized, and its crash recovery logic has failed in the past. I'm not even sure payday could actually be parallelized. However I think it is possible to make it fast enough so that restarting from the start after a crash isn't be a problem.

[Changaco]

I have rebased the payday rewrite branches on master, and I think they're ready for final review now.

[chadwhitacre]

Awesome, @Changaco, looking at them now. IRC

[chadwhitacre]

@Changaco Does it make sense to address #2443 in the rewrite?

[chadwhitacre]

@Changaco It looks like 97aacd0 is close to #2443, anyway.

[Changaco]

@whit537

It looks like 97aacd0 is close to #2443, anyway.

It was not my intention to fix #2443 in that commit, but it's true that it creates a one-to-one link between Gittip's exchanges and Balanced's transactions by inserting exchange_id in the meta field of the transaction. Is that sufficient to close #2443 or do we need more?

[chadwhitacre]

Hmm ... could be. I think we'll end up wanting to store some information locally to avoid expensive HTTP calls for things like #1681. No?