run Gratipay 156 #3486

Closed
whit537 opened this Issue May 27, 2015 · 67 comments

Projects

None yet

2 participants

@whit537
Member
whit537 commented May 27, 2015

155

@whit537 whit537 added the ★★★ label May 27, 2015
@whit537
Member
whit537 commented May 27, 2015

Don't forget to migrate tips to subscriptions for newly accepted teams!

@whit537 whit537 referenced this issue in gratipay/inside.gratipay.com May 28, 2015
Closed

Radar 8 #209

@whit537
Member
whit537 commented May 28, 2015

Teams reviewed (gratipay/inside.gratipay.com#210), still need to migrate tips to subscriptions.

@whit537
Member
whit537 commented May 28, 2015

I decided to go back and review users according to our old pattern, now that we're more confident that we don't have any more payout leaks (cf. #3453). However, there is a bug in the user review dashboard, and I've been setting everyone to suspicious instead of not suspicious. 😞

@whit537
Member
whit537 commented May 28, 2015

I'm going to break for lunch and coffee, and return to fix that bug and repair the database. Then I'll look at #3473.

@whit537
Member
whit537 commented May 28, 2015

Reticketed review bug as #3496. We are only looking at balanced_customer_href on the user dashboard. We should be looking at Braintree as well as PayPal. Let's not block this payday on improvements to user review. We can let user review slide for another week.

@whit537
Member
whit537 commented May 28, 2015

I'm thinking about how to repair the database, at least.

@whit537
Member
whit537 commented May 28, 2015

This selects all 34 relevant events:

  select ts, payload, payload->'id'
    from events
   where ts > '2015-05-28 17:00'::timestamptz and ts < '2015-05-28 18:00'::timestamptz
order by ts desc
   limit 100;
@whit537
Member
whit537 commented May 28, 2015

I'm going to take the database backup, and then run:

update participants
   set is_suspicious=null
 where id in (

      select payload->'id'
        from events
       where ts > '2015-05-28 17:00'::timestamptz and ts < '2015-05-28 18:00'::timestamptz

);

delete
  from events
 where ts > '2015-05-28 17:00'::timestamptz and ts < '2015-05-28 18:00'::timestamptz
@whit537
Member
whit537 commented May 28, 2015

Is that so wrong?

@whit537
Member
whit537 commented May 28, 2015

T-3 hours

@whit537
Member
whit537 commented May 28, 2015

I grabbed usernames with:

\copy (select username from participants where id in (select (payload->>'id')::int 
from events
where ts > '2015-05-28 17:00'::timestamptz and ts < '2015-05-28 18:00'::timestamptz)) 
to 'is_NOT_suspicious.csv' with csv
@whit537
Member
whit537 commented May 28, 2015

I set is_suspicious to false through the API:

#!/usr/bin/env python
import os, requests, sys

user_id = os.environ['GRATIPAY_USER_ID']
api_key = os.environ['GRATIPAY_API_KEY']

for line in sys.stdin:
    username = line.strip()
    url = 'https://gratipay.com/{}/toggle-is-suspicious.json'.format(username)
    response = requests.post(url, data={'to': 'false'}, auth=(user_id, api_key))
    print response.status_code, url

Two were toggled to true. I believe I had fixed them up in the UI before running the script, but I still don't understand why the script set them to true. I fixed them back up through the UI, and confirmed that all are now false with:

select username, is_suspicious
  from participants
 where id in (

    select (payload->>'id')::int
      from events
     where ts > '2015-05-28 17:00'::timestamptz and ts < '2015-05-28 18:00'::timestamptz

);
@whit537
Member
whit537 commented May 28, 2015

Okay! Back to payday ...

@whit537
Member
whit537 commented May 28, 2015

T-2.5 hours

@whit537
Member
whit537 commented May 28, 2015

Don't forget to migrate tips!

Last time: #3399 (comment).

@whit537
Member
whit537 commented May 28, 2015

T-1 hour.

@rohitpaulk I think we should bump payroll (#3433) to next week, and focus on Braintree (#3287) for today.

@whit537
Member
whit537 commented May 28, 2015

I'm going to review and probably merge #3470, then land #3473 and ... go for it ... hopefully ... 💐

@whit537
Member
whit537 commented May 28, 2015

Not forgetting to merge tips!

@rohitpaulk
Member

Sorry I was late -

@rohitpaulk I think we should bump payroll (#3433) to next week, and focus on Braintree (#3287) for today.

Yep. Braintree is an external deadline, payroll is in our control :)

@whit537
Member
whit537 commented May 28, 2015

@rohitpaulk Want to work on converting tips to subscriptions for new teams since last week?

@rohitpaulk
Member

Sure, on it.

@whit537
Member
whit537 commented May 28, 2015

!m @rohitpaulk

We've missed the bank payout deadline for today. We have so few users and everything is so disrupted, we can probably get away with slipping a few payouts until Monday. :-(

Let's land Braintree!

@whit537
Member
whit537 commented May 28, 2015

@rohitpaulk Can you paste your script for migrating tips for the new teams since last week?

@rohitpaulk
Member
from gratipay import wireup
from gratipay.models.team import Team, AlreadyMigrated

db = wireup.db(wireup.env())

new_teams = db.all("""
    SELECT slug
      FROM teams
     WHERE is_approved IS TRUE
""")

for slug in new_teams:
    team = Team.from_slug(slug)
    try:
        team.migrate_tips()
        print("Migrated tips for '%s'" % slug)
    except AlreadyMigrated:
        print("'%s' already migrated." % slug)

print("Done.")
@rohitpaulk
Member
'introtopython' already migrated.
'duo' already migrated.
'Gratipay' already migrated.
'sudo-room' already migrated.
'nuvola-player' already migrated.
'cyberpipe' already migrated.
'mojolicious' already migrated.
Migrated tips for 'pageres'
Migrated tips for 'sublimelinter'
Migrated tips for 'citizens-cooperative'
'catapultpgh' already migrated.
Migrated tips for 'sudomesh'
Migrated tips for 'perl-maven'
Migrated tips for 'exercism'
Done.
@whit537
Member
whit537 commented May 29, 2015

Cool. Is that in production?

(P.S. A couple minor style suggestions ... )

from gratipay.wireup import db, env
from gratipay.models.team import Team, AlreadyMigrated

db = db(env())

new_teams = db.all("""
    SELECT teams.*::teams
      FROM teams
     WHERE is_approved IS TRUE
""")

for team in new_teams:
    try:
        team.migrate_tips()
        print("Migrated tips for '%s'" % slug)
    except AlreadyMigrated:
        print("'%s' already migrated." % slug)

print("Done.")
@rohitpaulk
Member

How come 'catapultpgh' was already migrated?

@rohitpaulk
Member

Oh. I forgot that we'd turned on subscriptions.

@rohitpaulk
Member

Is that in production?

Yep, was on production.

@whit537
Member
whit537 commented May 29, 2015

Oh. I forgot that we'd turned on subscriptions.

Oooh, good point. So what's our test for AlreadyMigrated?

@rohitpaulk
Member

Whether subscription(s) exist with a ctime lesser than the team's ctime?

@whit537
Member
whit537 commented May 29, 2015

I don't think that works: doesn't the team have to be created before it can receive subscriptions?

@rohitpaulk
Member

When we migrate tips -> subscriptions, we copy the ctime as is...

@whit537
Member
whit537 commented May 29, 2015

Ah, gotcha. Wfm.

@rohitpaulk
Member

#3497 deployed, trying again..

@rohitpaulk
Member
'introtopython' already migrated.
'duo' already migrated.
'Gratipay' already migrated.
'sudo-room' already migrated.
'nuvola-player' already migrated.
'cyberpipe' already migrated.
Migrated tips for 'mojolicious'
'pageres' already migrated.
'sublimelinter' already migrated.
'citizens-cooperative' already migrated.
Migrated tips for 'catapultpgh'
'sudomesh' already migrated.
'perl-maven' already migrated.
'exercism' already migrated.
Done.
@whit537
Member
whit537 commented May 29, 2015
@whit537
Member
whit537 commented May 29, 2015

Alright! So ... we have tips migrated to subscriptions. We've merged #3470, and are just about done with #3473. What's next?

@whit537
Member
whit537 commented May 29, 2015

Droplet spun up and updated.

@whit537
Member
whit537 commented May 29, 2015

Backup started (now that #3473 is done) ...

@whit537
Member
whit537 commented May 29, 2015

Backup completed and verified.

@whit537
Member
whit537 commented May 29, 2015

Alright. Time for our first charge on Braintree. Here goes nothing ...

@whit537
Member
whit537 commented May 29, 2015

Script started.

@whit537
Member
whit537 commented May 29, 2015

Holds are happening ...

@whit537
Member
whit537 commented May 29, 2015

#3291 sure is annoying ...

@whit537
Member
whit537 commented May 29, 2015

Captures are happening ...

@whit537
Member
whit537 commented May 29, 2015

Captured 86 card holds.

@whit537
Member
whit537 commented May 29, 2015

Did payout for 7 participants.

@whit537
Member
whit537 commented May 29, 2015

This is going quickly!

@whit537
Member
whit537 commented May 29, 2015

Script ran for a minute (0:01:43.621884).

@whit537
Member
whit537 commented May 29, 2015

O.O

@whit537
Member
whit537 commented May 29, 2015

screen shot 2015-05-29 at 2 27 46 am

@whit537
Member
whit537 commented May 29, 2015

screen shot 2015-05-29 at 2 28 39 am

@whit537
Member
whit537 commented May 29, 2015

Good effing work, @rohitpaulk. Looks like you nailed it. 🔨 💅

@whit537
Member
whit537 commented May 29, 2015

Now next week we gotta get you paid again. :)

@whit537
Member
whit537 commented May 29, 2015

Payouts look good in Balanced, no charges there.

@rohitpaulk
Member

Already? Man that's quick. 😃

@rohitpaulk
Member
@whit537
Member
whit537 commented May 29, 2015

Hit a bug in MassPay ...

Traceback (most recent call last):
  File "./bin/masspay.py", line 304, in <module>
    main()
  File "./bin/masspay.py", line 297, in main
    compute_output_csvs()
  File "./bin/masspay.py", line 174, in compute_output_csvs
    payees = [Payee(rec) for rec in csv.reader(open(INPUT_CSV))]
  File "./bin/masspay.py", line 62, in __init__
    self.gross = D(amount)
  File "/Users/whit537/lib/python2.7/decimal.py", line 547, in __new__
    "Invalid literal for Decimal: %r" % value)
  File "/Users/whit537/lib/python2.7/decimal.py", line 3873, in _raise_error
    raise error(explanation)
decimal.InvalidOperation: Invalid literal for Decimal: ''
@whit537
Member
whit537 commented May 29, 2015

I think it's because we're allowing 'null' fee_cap for PayPal?

@whit537
Member
whit537 commented May 29, 2015

Repaired with:

update exchange_routes set fee_cap=20 where network='paypal' and fee_cap is null;
@whit537
Member
whit537 commented May 29, 2015

42 results

@whit537
Member
whit537 commented May 29, 2015

Log downloaded and droplet destroyed.

@whit537
Member
whit537 commented May 29, 2015

MassPay done and posted back for 7 users.

@whit537
Member
whit537 commented May 29, 2015

Escrow shuffled.

@whit537
Member
whit537 commented May 29, 2015

Logs uploaded in gratipay/logs@91d80c9.

@whit537
Member
whit537 commented May 29, 2015

Thaaaaaaaaat is satisfying. Awesome work, @rohitpaulk. I will catch you flip side. 🌝

@whit537 whit537 closed this May 29, 2015
@whit537 whit537 added the Payday label Jan 21, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment