Improve telemetry (clarity and design)

[jamesefhawkins]

Context

PostHog's strategy is to focus everything we have for at least the next 6-12 months (and likely, beyond that), on making the Community Edition awesome and very popular.

PostHog currently uses PostHog to track itself where users haven't opted themselves out.

We believe tracking Daily Active Users is the most meaningful top metric to check if we are achieving our strategy. That's because it's the combination of (i) new users signing up and (ii) existing users sticking around.

We currently have basic telemetry in the product for non-opted out users (anyone can opt out at any time). We use PostHog to track the following:

• new PostHog user signed up - name, email and company name entered
  • = used to produce unique sign ups
• pageviews for PostHog users when they come back to the product
  • = used to produce DAUs

The challenge

There is no fine grained control for users over this telemetry:

• You can opt out of nothing or everything, and the opt out is on the /settings page where most users go to set up the product
• If a single user opts out, they opt out of (i) aggregated data (ii) identifiable data (ii) their entire team

This means almost every engaged user that we know of anecdotally has (understandably) opted out of our tracking.

We also are not clear over exactly how our telemetry works (we have just moved docs and need to add a new section for this - legacy explanation), and should clarify things. For example, if it is left on, we do not collect users' user data, which is probably important for people to know :)

Proposal

I would like us to (i) make it easier to remove personally identifiable information from our tracking but (ii) make it more likely people leave anonymized "counter" style tracking to let us assess page views properly.

There are several conditions that we would need to meet for this to be responsible:

• If a user is opted out, that user shall forever remain opted out of everything
• We should explain clearly in our new docs what the telemetry does
• We should raise this in the community first to give people a chance to talk about it

To solve this, I'd suggest:

1. we create a page somewhere that is specific to our telemetry in the app, as the regular settings page is becoming complex Ie setup>advanced.

• That page makes clear exactly what is tracked and what isn't.
• This page lets you opt out of the data categories we track in a more fine-grained way:
  • Team level (ideally this would only come from an admin role)
  • User and field level

2. We track anonymized aggregated usage information by default, but provide a code level change to manage this, that we would explain how to do in the docs. Just the minimum info would be collected to work out anonyized DAUs. I'd see that as the only cost of the free software, which helps us keep raising money and building cool stuff.

Why I think this is the right direction

If PostHog can track DAUs, we're much better able to explain the software's popularity to investors. There is significant social good in this - it lets us build a powerful, stable and well supported product and community.

However, our mini group of employees are not the only people in the community and I want to get feedback from anyone reading this before implementing it. Do you agree? Else, could you suggest a different approach to the challenge above?

[CaseGuide]

I saw the "opt out" button and the only reason I didn't click it was because I like you guys personally. I recalled reading somewhere that you didn't use our data, so I would've opted out even knowing that. Not using our customer data is an expectation/feature of the product.

Otherwise I will always opt out of everything. Even if control is fine grained.

I may not be the best representative user because I also browse the web with Chrome/Firefox always incognito w/near max privacy settings (except JS allowed) + ublock, so get other users feedback.
Tracking a good KPI (like DAUs) is critical to the success of your company. You don't need to track what all users do, only a representative set, but some basic engagement tracking like sign ins and amount of time used in exchange for using your free product is a reality I'm cool with and I don't think will cost you any users.

[timgl]

The other problem we have is that it's really hard to apply the opt out to an entire instance (especially for our Python events) when settings are applied to a team/person. This could inadvertently lead to people who really care about the tracking being disabled still sending events out.

I suggest instead we move the entire opt-out to a boolean setting when you run the server. That way people who care can make sure they send absolutely nothing.