run Gratipay 194

[clone1018]

#500

[clone1018]

Having an issue pulling down from GitHub atm, debugging

[clone1018]

$ cat migrate-tips.log
Migrated 1 tip(s) for '<removed>'
Done.

@rohitpaulk is this expected?

[clone1018]

Backup completed and verified.

[clone1018]

payday.sh is still running but the log file hasn't updated in awhile...?

[clone1018]

Yeah, payday.sh is hung. cc @whit537 @rohitpaulk

Current status is capturing cards

[clone1018]

Here's the partial log file https://github.com/gratipay/logs/blob/master/payday/gratipay-194.log-partial

[chadwhitacre]

Maybe @rohitpaulk is incommunicado?

@clone1018 Sec, digging up previous payday ticket ...

[chadwhitacre]

It had to do with a network error leading to a wedged process in Postgres ...

[chadwhitacre]

I think it was the first week that @rohitpaulk ran payday last month ...

[chadwhitacre]

Here it is, @clone1018: #477 (comment).

Any update on our situation? How can I help?

[chadwhitacre]

This could be a red herring, but I did make a seemingly minor configuration change at Mandrill: I verified our domain. I mention it because it looks from the log like we made it through card captures, but stopped short of sending notifications (I didn't receive one, e.g.).

[chadwhitacre]

The log ends with a capture. Did we make it through all captures?

[clone1018]

@whit537 we did 59 captures last week and 69 so far in the partial log

[clone1018]

I'm not sure how to check how many captures we should have done

[chadwhitacre]

Well, it's okay to rerun because the captures shouldn't be recaptured. I was asking more to say—where did we actually die? And why?

[clone1018]

I have no idea, straced the process at the end and it was just hung. I'll return and monitor with strace to see what it's hanging on

[clone1018]

pid-3383 thread-139848410195712 (MainThread) Traceback (most recent call last):
pid-3383 thread-139848410195712 (MainThread)   File "/home/clone1018/Projects/Gratipay/gratipay.com/gratipay/cli.py", line 24, in payday
pid-3383 thread-139848410195712 (MainThread)     Payday.start().run()
pid-3383 thread-139848410195712 (MainThread)   File "/home/clone1018/Projects/Gratipay/gratipay.com/gratipay/billing/payday.py", line 132, in run
pid-3383 thread-139848410195712 (MainThread)     self.payin()
pid-3383 thread-139848410195712 (MainThread)   File "/home/clone1018/Projects/Gratipay/gratipay.com/gratipay/billing/payday.py", line 153, in payin
pid-3383 thread-139848410195712 (MainThread)     holds = self.create_card_holds(cursor)
pid-3383 thread-139848410195712 (MainThread)   File "/home/clone1018/Projects/Gratipay/gratipay.com/gratipay/billing/payday.py", line 215, in create_card_holds
pid-3383 thread-139848410195712 (MainThread)     holds = self.fetch_card_holds(participant_ids)
pid-3383 thread-139848410195712 (MainThread)   File "/home/clone1018/Projects/Gratipay/gratipay.com/gratipay/billing/payday.py", line 196, in fetch_card_holds
pid-3383 thread-139848410195712 (MainThread)     cancel_card_hold(hold)
pid-3383 thread-139848410195712 (MainThread)   File "/home/clone1018/Projects/Gratipay/gratipay.com/gratipay/billing/exchanges.py", line 147, in cancel_card_hold
pid-3383 thread-139848410195712 (MainThread)     assert result.is_success
pid-3383 thread-139848410195712 (MainThread) AssertionError
pid-3321 thread-140454600414976 (MainThread) exchange 80448 has an unexpected status: pre

[clone1018]

Did get a

Script ran for a minute (0:01:34.523306). though

[chadwhitacre]

Interesting!

[clone1018]

I see $1152 posted on Braintree, that seem about average?

[chadwhitacre]

@clone1018 Yeah, seems likely that captures worked. Did we hang up in a Postgres call? Did you follow the #477 (comment) thread at all?

[clone1018]

It could have been related to the network, I do Payday on my laptop on an aircard but it seemly worked fine the past two times and I don't see any obvious errors?

[clone1018]

I can do the droplet thing in the future if this doesn't work

[chadwhitacre]

[chadwhitacre]

#508 (comment) indicates that exchange 80448 is what we died in the middle of.

[clone1018]

@whit537 there's an SSL error in 80448's exchange_routes

[chadwhitacre]

There we go. Sounds like a network connectivity issue in an API call to Braintree. The little-used pre state is a strong clue as well.

[chadwhitacre]

So I guess we need to make sure a) that everyone after 80448 worked properly, and b) that whoever 80448 was for had a second charge in the re-run. Sound right?

[chadwhitacre]

worked properly

In the re-run.

[chadwhitacre]

I moved the error into the exchanges table for 80448 like so:

=> update exchanges set note='SSLError(SSLError(SSLEOFError(8, u''EOF occurred in violation of protocol (_ssl.c:590)''),),)', status='failed' where id=80448;

[clone1018]

Looks like we have 6's None's this week. #485 (comment)

[chadwhitacre]

We want to run update_error to update giving, etc., so here's a script:

#!/usr/bin/env python2
from __future__ import absolute_import, division, print_function, unicode_literals

import sys

from gratipay import wireup
from gratipay.models.exchange_route import ExchangeRoute

wireup.db(wireup.env())

for route_id in sys.argv[1:]:
    ExchangeRoute.from_id(route_id).update_error('')

[clone1018]

=> update exchange_routes set fee_cap = 20.00 where fee_cap is null and network = 'paypal' and address in (...list);
UPDATE 6

[chadwhitacre]

This look right, @clone1018?

[gratipay] $ heroku config -s -a gratipay | run - ./env/bin/python clear-route-errors.py 9051 9136

Here's run.

Here's clear-route-errors.py:

from __future__ import absolute_import, division, print_function, unicode_literals

import sys

from gratipay import wireup
from gratipay.models.exchange_route import ExchangeRoute

wireup.db(wireup.env())

for route_id in sys.argv[1:]:
    ExchangeRoute.from_id(route_id).update_error('')
    print("cleared", route_id)

[chadwhitacre]

Local test looks okay:

[gratipay] $ run defaults.env local.env python clear-route-errors.py 2182
cleared 2182

[chadwhitacre]

Here goes ...

[clone1018]

Looks good to me

[chadwhitacre]

!m @clone1018

gratipay::MAROON=> select id, error from exchange_routes where id in (9051, 9136);
┌──────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│  id  │                                                                error
├──────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│ 9051 │ ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='api.braintreegateway.com', port=443): Read timed out. (read timeout=
│ 9136 │ SSLError(SSLError(SSLEOFError(8, u'EOF occurred in violation of protocol (_ssl.c:590)'),),)
└──────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
(2 rows)

[gratipay] $ heroku config -s -a gratipay | run - ./env/bin/python clear-route-errors.py 9051 9136
cleared 9051
cleared 9136
[gratipay] $

gratipay::MAROON=> select id, error from exchange_routes where id in (9051, 9136);
┌──────┬───────┐
│  id  │ error │
├──────┼───────┤
│ 9051 │       │
│ 9136 │       │
└──────┴───────┘
(2 rows)

[clone1018]

OMG for the first time in my payday experience, fee's and totals match up on the first try!

[chadwhitacre]

I think you've earned it this week, @clone1018. :-)

[clone1018]

POSTed MassPay back to Gratipay for 51 users.

[clone1018]

Verified:

=> select count(*) from exchanges where DATE(timestamp) = '2016-02-18' and recorder = 'clone1018';
 count 
-------
    51

[clone1018]

Masspay: https://github.com/gratipay/logs/commit/c1e846769eaef56677f38f562a9df01ff94650d2
Payday: https://github.com/gratipay/logs/commit/956b9a96c68dc30aa89756c249dfd927c9de9a76

[clone1018]

[clone1018]

Decline rate = 13.92

EEP!

[chadwhitacre]

Users notified:

https://gratipay.freshdesk.com/helpdesk/tickets/4133
https://gratipay.freshdesk.com/helpdesk/tickets/4134

[clone1018]

1152 * 5 = 5760, paypal says we have 4,077.82, gonna add $2000?

[chadwhitacre]

@clone1018 Sounds good. I'm winding down hear, gonna go find a bus. I will try to check in later tonight but I might not get to until tomorrow.

Good work! :-)

!m @clone1018

[clone1018]

!m @whit537 thanks for the help with the explosion!

[clone1018]

You just added money from your bank account
The money will be transferred from your bank account to your PayPal account in 3-5 business days.

Added $2000USD

[clone1018]

Interesting note about https://github.com/gratipay/logs/commit/956b9a96c68dc30aa89756c249dfd927c9de9a76#diff-094c93b75ab960c9d1f51dcb9ec35469R168

I didn't see that in my shell, maybe when I was running payday.sh stderr went nowhere?

[clone1018]

I think we're done, I've finished all the normal tasks, gonna leave open until @whit537 confirms his cleanup is done

[chadwhitacre]

My cleanup is done.

!m @clone1018

[clone1018]

Done

[rohitpaulk]

!m @whit537 @clone1018. I forgot about Payday 😑, I've set a recurring reminder, hopefully won't happen again :)

[chadwhitacre]

No worries @rohitpaulk. As long as you don't forget and everything doesn't explode during #314! :)