Join GitHub today
16 Ways to Measure Network Effects -- Andressen Horowitz #29
Update as of 2019-02-04:
I had a meeting with Jesse Walden ( https://www.linkedin.com/in/jessewalden ) of a16z crypto a couple weeks ago. He asked me to read https://a16z.com/2018/12/13/16-metrics-network-effects/ .
They invest in companies that have
We definitely fit in (1), but we need to do some data sleuthing on the network effects stuff.
From this blog post he sent me, we need to know the following. We should find a way to triage this data report and create something thats representative of the project's numbers, so we can go back to him and get advice from a16z crypto on areas to focus in Q1.
#1 Organic vs. paid users
Google Analytics reports ~20% of new users and ~18% of visits being organic from Jan 2018 to Jan 2019, but because GA has a unique definition of "users", this metric has questionable accuracy due to potential double counting (if a user comes back on a different browser, etc, they would be counted twice).
#3 Time series of paid CAC
The $6-$8 CAC previously reported is not completely true, and we shouldn't be anchoring off of it. Of the paid acquisition channels we were going for, facebook seemed to be the most effective. Our data from 2018 July to 2018 November shows that over time the cost per lead was ~$23, ~$9.50, ~$8.50, to $14 - there doesn't seem to be data in September. Facebook ads have been stopped as of 2018 December.
Twitter and Stackoverflow yielded even worse results and higher cost per lead, so those pursuits were discontinued as well.
The cost per lead numbers are worsened by the fact that these dollar calculations are only to get a sign-up - we weren't tracking to see if the sign-ups converted and did a bounty.
Dean has provided Facebook login creds to see if we can connect Metabase data to get actual bounty completions off the CPC. My hunch is that if just generating signups/leads is already high, it'll be even higher once we factor in who did a bounty, and also by country (India was much cheaper), so I would say priority on setting that integration up is lower.
#6 User retention cohorts
Retention chart is defined as cohorts who join, and then perform a contributor or funder related action 0, 1, 2, 3, etc months later.
#8 Dollar retention & paid user retention cohorts
A Metabase query is up for this, but because we don't have revenue, I'd like to double check these calculations to ensure I'm understanding them correctly. They may not be necessarily relevant in our case.
#8a Segment by behavioral cohorts
#9 Retention by location/geography
#11 Match rate (aka utilization rate, success rate, etc.)
#12 Market depth
#13 Time to find a match (or inventory turnover, or days to turn)
#14 Concentration or fragmentation of supply and demand
#15 Pricing power
#16 Unit economics
Dashboard as of 2018-01-10:
I'd add caution here that in actuality, our LTV is effectively $0 (for bounties) because we are in the process of validating / testing the proposition of a % cut. So the calculations above are theoretical.
Given we're not doing any marketing spend right now, 100% of our new users are organic, right? The problem is this isn't huge growth in absolute terms thus far, though it's not bad. Not sure how GA is coming up with the 20% figure.
Most of our CAC has been aimed at the contributor, traditionally, given we want to find folks who will do a great job. These numbers start to paint a picture in the other direction -- where we should be aiming to find funders who are willing to pay for bounties.
This side, in OSS, may be hard to find. Yet it's where the LTV lies.
hmm this metric seems low. i think you mean that 20% of these users were from organic search. whereas organic acquisition is all users we didnt pay for!
Organic Acquisition channels == Anything but paid search. So actually I think the number is closer to 78% per my reading of the acquisition stats in GA.
I agree that the CAC for all of 2018 was much higher than $8.
I guess the question is -- what CAC do we report? We tested a lot of different targets/creatives over time, and I think $6 CAC is the best we got. It's probably safe to assume we'd launch new paid campaigns on customer segments, and with marketing that targeted our most efficient campaigns.
I think we should be able to do this moving forward.. We track the utm variables in the DB now.
Maybe its worth scheduling a brainstorming here?
Super interesting.. So for every 100 new users, we eventually have one 1 user who stays active consistently forever.. (er... not forever, but for a while)
Worth a brainstorm here too.
Do u mean 14-18 days of the month ?
This is really confusing to me. I see what you're saying about 'organic search'. 78% could be a closer, but the way Google Analytics tracks the term 'users' is technically not unique. I'll see if I can confirm the rough 78% number by patching together Dean's aggregated data, and user join dates from Gitcoin. edit update: Doing some back of the napkin math using Dean's results on Facebook acquisitions and our newly acquired unique (no thanks GA) users, comes out to around 80.5%.
I would recommend reporting CACs by channels/cohort, maybe only Facebook and by quarter? Because they're variable and we've cut off marketing, I don't think reporting one averaged metric makes sense.
Yep, I actually have some queries in Metabase up using that data. Specifically, I don't think we can (or have yet) back-filled that data for the past (2017 and most of 2018).
Yeah, roughly. This is the asymptote of the retention curve. For every cohort, the goal is to not have as many people drop off at every stage, and hopefully have a higher baseline.
Yes, you're right, week -> month. Critical typo
Happy to jam. (though it sounds like you maybe came to a conclusion from the rest of the paragraph above?)
this is a good idea.. if i give you access to the gdoc for doing projections, can you put some of the cohorts into it? (tracking here #37 )