Skip to content
This repository has been archived by the owner. It is now read-only.

run Gittip 101 #2362

Closed
chadwhitacre opened this issue May 8, 2014 · 89 comments
Closed

run Gittip 101 #2362

chadwhitacre opened this issue May 8, 2014 · 89 comments

Comments

@chadwhitacre
Copy link
Contributor

@chadwhitacre chadwhitacre commented May 8, 2014

100

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

Reviewed 62 accounts. Left 3 unreviewed (see #2341).

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

Backup taken and verified.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

Droplet spun up and updated.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

Script started.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

Traceback (most recent call last):
  File "./env/bin/payday", line 9, in <module>
    load_entry_point('gittip==10.2.0-dev', 'console_scripts', 'payday')()
  File "/home/whit537/www.gittip.com/gittip/cli.py", line 17, in payday
    db = wireup.db()
  File "/home/whit537/www.gittip.com/gittip/wireup.py", line 33, in db
    psycopg2.extras.register_hstore(cursor, globally=True, unicode=True)
  File "/home/whit537/www.gittip.com/env/local/lib/python2.7/site-packages/psycopg2/extras.py", line 775, in register_hstore
    "hstore type not found in the database. "
psycopg2.ProgrammingError: hstore type not found in the database. please install it from your 'contrib/hstore.sql' file

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

I updated the droplet improperly. I forgot to pull before making.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

Script restarted.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

pid-1657 thread-140589118981888 (MainThread) Traceback (most recent call last):
pid-1657 thread-140589118981888 (MainThread)   File "/home/whit537/www.gittip.com/gittip/cli.py", line 32, in payday
pid-1657 thread-140589118981888 (MainThread)     Payday(db).run()
pid-1657 thread-140589118981888 (MainThread)   File "/home/whit537/www.gittip.com/gittip/billing/payday.py", line 140, in run
pid-1657 thread-140589118981888 (MainThread)     self.pachinko(ts_start, self.genparticipants(ts_start, ts_start))
pid-1657 thread-140589118981888 (MainThread)   File "/home/whit537/www.gittip.com/gittip/billing/payday.py", line 264, in pachinko
pid-1657 thread-140589118981888 (MainThread)     tip(take['member'], amount)
pid-1657 thread-140589118981888 (MainThread)   File "/home/whit537/www.gittip.com/gittip/billing/payday.py", line 258, in tip
pid-1657 thread-140589118981888 (MainThread)     , pachinko=True
pid-1657 thread-140589118981888 (MainThread)   File "/home/whit537/www.gittip.com/gittip/billing/payday.py", line 424, in tip
pid-1657 thread-140589118981888 (MainThread)     tip['amount'], pachinko=pachinko):
pid-1657 thread-140589118981888 (MainThread)   File "/home/whit537/www.gittip.com/gittip/billing/payday.py", line 456, in transfer
pid-1657 thread-140589118981888 (MainThread)     self.credit_participant(cursor, tippee, amount)
pid-1657 thread-140589118981888 (MainThread)   File "/home/whit537/www.gittip.com/gittip/billing/payday.py", line 507, in credit_participant
pid-1657 thread-140589118981888 (MainThread)     assert rec is not None, (participant, amount)  # sanity check
pid-1657 thread-140589118981888 (MainThread) AssertionError: (u'aaronwinborn', Decimal('0.01'))

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

pending is NULL for @aaronwinborn.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

Why is pending NULL?

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

Pending is NULL for 43972 participants.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

There are 73659 participants total.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

Maybe @aaronwinborn was added to the Drupal team during payday?

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

That's the sort of thing that happened the last time we had issues around pending, way back (will have to dig for that).

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

ctime is 2014-05-02 05:57:04.323376+00. Is that during last week's payday?

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

Last payday (#2340) was 2014-05-01.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

The only place where pending is explicitly set to NULL is in clear_pending_to_balance, which happens right after pachinko.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

But I seem to recall that NULL + 0 results in a NULL, and that we've had that happen before.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

=> select (0 + null);                                                                     ┌──────────┐
│ ?column? │
├──────────┤
│        ¤ │
└──────────┘
(1 row)

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

Sanity check:

=> select (null + 0);
┌──────────┐
│ ?column? │
├──────────┤
│        ¤ │
└──────────┘
(1 row)

@Changaco
Copy link
Contributor

@Changaco Changaco commented May 8, 2014

Why is pending allowed to be NULL? 0 seems like a sane default to me.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

@Changaco We use NULL as a sentinel value during payday.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

Okay! @aaronwinborn hasn't joined Gittip. :-)

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

So the question is, how is aaronwinborn part of the DrupalMedia team without being a member of the site?

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

Okay! So, aaronwinborn's participant record was created at 2012-07-11 14:48:30.679927+00, which is before we started using randomized usernames for stub participants. The bug is that stub participants can be added as members of a team. A quick query of username and ctime shows that 2012-09-13 is the cutoff. That was the date we started using randomized usernames for stub participants.

Precluding adding stub participants as team members ticketed as #2364.

Randomizing usernames for old stub participants reticketed as #2365.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 8, 2014

So what's the fix here? We should delete from takes where the user is a stub participant.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 13, 2014

That's out of scope here, anyway. That's not a bug we need to fix or repair here.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 13, 2014

We've so far identified two causes of duplicate transfers:

  • A participant takes from a team that also gives to them.
  • Payday crashed and we reran it with a bug.

Does that account for all of the duplicates we're seeing?

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

Let's assume for the moment that all of the duplicate transfers before last payday are due to taking from a team that gives to you. Let's look at just the duplicates from last payday. Let's assume there are up to three amongst the 66 that are also legitimate. So we have 63-66 bad duplicates. Here's the question (IRC): What are the dollar amounts involved?

Turns out there are only five duplicates that are $10 or more. They're all on the Gittip team. The sixth-highest (at $4) is also from Gittip. Here's a breakdown:

41   0.01
20   0.02 -  9.99
 5  10.00 - 50.00

There are 23 teams affected by this. It's really only the Gittip team where the dollar amounts are at all problematic. One option is to simply say that this was a mistake and let the teams eat it. Right now the teams are eating it. Some team members received more than their fair share. Except for Gittip, the amounts are so low and the teams are so barely active that it's really not a big deal.

Since we are Gittip :-) we can ask ourselves: are we okay with 12 of our members getting a bonus last week? Here are the people in question and the amounts of their bonuses:

=> select tippee, array_agg[1] from bar where count > 1 and date = '2014-05-08'::date and tipper='Gittip' order by tipper, array_agg;
┌─────────────┬───────────┐
│   tippee    │ array_agg │
├─────────────┼───────────┤
│ thefoxis    │      0.01 │
│ mtrythall   │      0.01 │
│ Aaron1011   │      0.01 │
│ galuszkak   │      0.02 │
│ dominic     │      1.00 │
│ pjf         │      2.00 │
│ ESWAT       │      4.00 │
│ sim6        │     10.00 │
│ duckinator  │     16.00 │
│ seanlinsley │     18.01 │
│ Changaco    │     20.00 │
│ patcon      │     50.00 │
└─────────────┴───────────┘
(12 rows)

Personally I'm fine giving these folks a bonus. I think that makes more sense than spending more time trying to reverse this. In which case, we can call it case closed on the assessment and db repair, and move on to fixing the bugs that landed us here in the first place. Any objections?

1087359942_1372269225

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

I'm going to proceed on that basis.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

Object if you want to! :-)

@patcon
Copy link
Contributor

@patcon patcon commented May 14, 2014

Cool. I'd like to do a one-time split with @rummik, if that's cool. I've just adjusted Kim's amount accordingly, and I'll adjust back after payday :)

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

Now I'm looking at this constrained by as_team_member. I know we added that fairly recently. I don't know if we tried to set it to true for transfers in the past. It's at least useful for this week.

I'm writing up a blog post about this, somewhere to point people if/when they notice.

select id, timestamp::date as date, tipper, tippee, amount 
into temp table foo from transfers where as_team_member;

select date, tipper, tippee, count(id), array_agg(amount)
into temp bar from foo group by date, tipper, tippee order by count;

select * from bar where count > 1 order by date;

select date, count(date) as nduplicates from bar
where count > 1 group by date order by date;

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

Awesome, thanks @patcon! :-)

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

=> select tipper, sum(array_agg[1]) from bar where count > 1 group by tipper order by sum; 
┌────────────────┬────────┐
│     tipper     │  sum   │
├────────────────┼────────┤
│ 3v4l.org       │   0.01 │
│ thechangelog   │   0.01 │
│ saidozcan      │   0.01 │
│ BrightworkCoRe │   0.01 │
│ NuvolaPlayer   │   0.01 │
│ goodcode       │   0.01 │
│ esdiscuss      │   0.01 │
│ by_codebar     │   0.01 │
│ promisejs.org  │   0.01 │
│ meteorjs       │   0.01 │
│ teampopong     │   0.02 │
│ GetStorage     │   0.02 │
│ Innov_Africa   │   0.02 │
│ Bountysource   │   0.04 │
│ IPvoices       │   0.04 │
│ Coackme        │   0.09 │
│ Shields        │   1.58 │
│ Aspen          │   3.50 │
│ UnissonCo      │   4.04 │
│ rubygems       │  10.83 │
│ Gittip         │ 143.05 │
└────────────────┴────────┘
(21 rows)

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

That's $163.33 total.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

Not sure why my result differs from @Changaco's: #2362 (comment).

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

The array_agg[1] is not stable. I was pulling @seanlinsley's 40.00 transfer instead of the 18.01 one.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

Now I'm at 141.34, which fits with @Changaco's earlier result.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

Now to determine which teams never got paid at all.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

This would be any teams after the first failure.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

Actually, any later members of the team we were processing when we crashed would also have not gotten a payout.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

Using select * from takes where team='DrupalMedia' order by ctime desc; on the backup (where I can still see aaronwinborn), there are two affected users on the DrupalMedia team.

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

This is also inferrable from their Members page:

screen shot 2014-05-14 at 4 20 30 pm

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

Okay, so how many teams are after the DrupalMedia team?

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

=> select count(id) from participants where claimed_time >= \
    (select claimed_time from participants where username='DrupalMedia');
┌───────┐
│ count │
├───────┤
│   561 │
└───────┘
(1 row)

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

=> select username from participants p join takes t on p.username = t.team 
where claimed_time >= (select claimed_time from participants where username='DrupalMedia') 
and number='plural' group by username;
┌───────────────┐
│   username    │
├───────────────┤
│ DrupalMedia   │
│ PerlDancer    │
│ PhongRobotics │
└───────────────┘
(3 rows)

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

@chadwhitacre
Copy link
Contributor Author

@chadwhitacre chadwhitacre commented May 14, 2014

Case closed.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants