run Gratipay 118

[chadwhitacre]

117

[chadwhitacre]

Reviewed 31 accounts.

[chadwhitacre]

Backup taken and verified.

[chadwhitacre]

Droplet spun up and updated.

[chadwhitacre]

Script started.

[chadwhitacre]

pid-1372 thread-139886181644032 (MainThread) Traceback (most recent call last):
pid-1372 thread-139886181644032 (MainThread)   File "/home/whit537/gratipay.com/gratipay/cli.py", line 29, in payday
pid-1372 thread-139886181644032 (MainThread)     Payday.start().run()
pid-1372 thread-139886181644032 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 115, in run
pid-1372 thread-139886181644032 (MainThread)     self.payin()
pid-1372 thread-139886181644032 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 146, in payin
pid-1372 thread-139886181644032 (MainThread)     self.settle_card_holds(cursor, holds)
pid-1372 thread-139886181644032 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 469, in settle_card_holds
pid-1372 thread-139886181644032 (MainThread)     threaded_map(capture, participants)
pid-1372 thread-139886181644032 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 31, in threaded_map
pid-1372 thread-139886181644032 (MainThread)     r = pool.map(func, iterable)
pid-1372 thread-139886181644032 (MainThread)   File "/usr/lib/python2.7/multiprocessing/pool.py", line 227, in map
pid-1372 thread-139886181644032 (MainThread)     return self.map_async(func, iterable, chunksize).get()
pid-1372 thread-139886181644032 (MainThread)   File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
pid-1372 thread-139886181644032 (MainThread)     raise self._value
pid-1372 thread-139886181644032 (MainThread) HTTPError: HTTPError(status='Conflict', category_code='hold-already-captured', additional=None, status_code=409, category_type='logical', extras={}, request_id='OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', response=None, description='This hold xxxxxxxxxxxxxxxxxxxxxxxx has already been captured. Your request id is OHMxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.')

[chadwhitacre]

IRC

[chadwhitacre]

Any help with this, @balanced?

[chadwhitacre]

Balanced IRC

[chadwhitacre]

Mail sent to support@balancedpayments.com:

Greetings! :-)

We're getting an unexpected "hold already captured." Here's the request:

https://dashboard.balancedpayments.com/#/marketplaces/MP12Xw5lL6iaILtqImIoroDL/logs/OHM7ffaa646345611e4ac8902b12035401b

Our best hypothesis at this point is that this is due to a concurrency bug on our side (last week we started parallelizing API requests during our payday). But if that's the case, why don't we see the capture that did succeed? E.g., here:

https://dashboard.balancedpayments.com/#/marketplaces/MP12Xw5lL6iaILtqImIoroDL/debits/WD4sOTuY5oHG9INnsJ1hqN7I

See also GitHub:

#2729 (comment)

And IRC:

https://botbot.me/freenode/balanced/2014-09-04/?msg=21025263&page=1

chad

[chadwhitacre]

Sounds from support convo w/ @msherry that this may be an issue on Balanced's side. The successful debit against the hold was in an inconsistent state (no logs or events in the dashboard, and not listed under payments on the customer page). @msherry reconciled it to a failed state so now we have the hold for the user and two failed debits against it.

[chadwhitacre]

Since we crashed while the mega-transaction was open, we are likely to hit #2663 again.

[chadwhitacre]

Thu Sep  4 20:03:06 UTC 2014
pid-1876 thread-139677348706048 (MainThread) Traceback (most recent call last):
pid-1876 thread-139677348706048 (MainThread)   File "/home/whit537/gratipay.com/gratipay/cli.py", line 28, in payday
pid-1876 thread-139677348706048 (MainThread)     sync_with_balanced(cursor)
pid-1876 thread-139677348706048 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/exchanges.py", line 361, in sync_with_balanced
pid-1876 thread-139677348706048 (MainThread)     record_exchange_result(db, e.id, status, error, p)
pid-1876 thread-139677348706048 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/exchanges.py", line 300, in record_exchange_result
pid-1876 thread-139677348706048 (MainThread)     with db.get_cursor() as cursor:
pid-1876 thread-139677348706048 (MainThread) AttributeError: 'SimpleNamedTupleCursor' object has no attribute 'get_cursor'

[chadwhitacre]

That's from rerunning with this hot-fix. IRC

[Changaco]

Let's drop the outer cursor for now, it's only an extra protection, it's not actually needed.

diff --git a/gratipay/cli.py b/gratipay/cli.py
index 0b101f5..4f0c12a 100644
--- a/gratipay/cli.py
+++ b/gratipay/cli.py
@@ -24,8 +24,7 @@ def payday():
     from gratipay.billing.payday import Payday

     try:
-        with db.get_cursor() as cursor:
-            sync_with_balanced(cursor)
+        sync_with_balanced(db)
         Payday.start().run()
     except KeyboardInterrupt:
         pass

[chadwhitacre]

Hot fix edited.

[chadwhitacre]

Rerunning ...

[chadwhitacre]

Script ran for 19 minutes (0:19:26.089372).

[chadwhitacre]

What's our check for #2663?

[chadwhitacre]

#2663 checked. Only one user affected, and the amount involved is pennies. Safely skipping.

[chadwhitacre]

https://twitter.com/Gratipay/status/507632502135988224 pinned

[chadwhitacre]

Log downloaded and droplet destroyed.

[chadwhitacre]

MassPay is off. Gonna bisect to find the record with the wrong fee cap.

[chadwhitacre]

Found it.

[chadwhitacre]

MassPay done and posted back for 63 users.

[chadwhitacre]

Escrow shuffled. I moved $50,000 to cold storage today (first time in a while since I hadn't checked in on New Alliance for a while).

[Changaco]

Stats for the Gratipay team: receiving amount is up by 5.3%, number of tippers is up by 13.8%.

[chadwhitacre]

😊