Metrics: Requirements for Analytics and Database

[adamlofting]

I'm closing the requirements gathering checklist #56 as we need to move on implementation.

Known tasks:

For @adamlofting and or dev support (TBC @simonwex)

• Setup a new GA property via MoCo for id.webmaker.org (Bug is here)
• Setup basic Google Analytics in the id.webmaker.org codebase (PR is open here #93)
  • Use react-ga module to do this
• Review the front-end experience and tech stack to see what tracking beyond basic pageviews is required
• Setup GA tracking of the in-page event tracking we care about (validation errors, field focus, etc) Ga #137
• Setup conversions Goals and funnels in the GA account
  • Account creation and login
  • Field by field funnel
• Ensure we're tracking pageload times as part of our GA configuration
• Solve / Spec how we identify traffic sources (teach / webmaker / other) within the GA reporting (this is covered by standard GA referral data)
• Solve / Spec how we update GA code, and reporting configuration across all other properties currently using login
• Not a blocker: Proof of concept implementation for Optimizely

Open Questions:

For @jbuck, @secretrobotron, or others in the know.

• Are we replacing the existing Users table that sits behind the current login platform?
  • This shouldn't block or slow down shipping, but will cause a bunch of knock-on work for our integration db and KPI dashboards
• Are we launching just with teach, or swapping out all instances of login at the same time?
• Did we preserve the existing work on the Referrer API and this, or has this been killed? Solved in Investigate login referrals #97
• How will we identify in the database, the difference between Webmaker Account Holders, and Webmaker Users? We need some kind of table or flag to say which sites the user has authenticated against. Example query I'd want to be able to run = show me all account holders who use teach.m.o but never webmaker.org. Solved in Investigate login referrals #97

Reporting Requirements:

For @adamlofting

For ID as a standalone Service

Conversion rates

• UV to Sign Up
• UV to Log In
  GA Goals Setup

Abandon rate

• % of users who start but never complete sign up
• % of users who start but never complete log in
• overall bounce rate - users who land on id.w.o but don't start sign up or log in
  These can be seen in standard GA reporting using the Goals

Abandon funnel

• % of users who exit the funnel at each stage of sign up (by view is fine, but by input is better)
• % of users who exit the funnel at each stage of log in (by view is fine, but by input is better)

Error frequency

• Break down of validation errors by message and both % and # of occurrences (this being recorded in GA)
• Break down of HTTP errors (404s, 500s, etc) created by both the API and WWW interface (TBC where we want to measure this)

For Products that use the ID Service (webmaker and teach.m.o)

• Need to see users who go out to id.w.o, and then return as part of a single session
• Before and after reporting of impact on existing tools and users
• Before and after reporting of email opt-in rate given the new UX (separating the two checkboxes)

[adamlofting]

Pings:

@jbuck @secretrobotron could you have a look at these open questions please?
@simonwex is there dev capacity here, or is it best I start hacking on this myself?
@thisandagain cc/ re: requirement to identify accounts by the sites they have used

[simonwex]

@secretrobotron, could we add this to the stack to consider tomorrow during our login triage?

[secretrobotron]

@simonwex 👍 Can remain in this milestone to review.

@adamlofting trying to answer q's. @jbuck can confirm:

Are we replacing the existing Users table

@jbuck's proposal was to keep the old table around, while using a new table to store new/updated users. So, both will be in use until a certain (yet unspecified) date.

Are we launching just with teach, or swapping out all instances of login at the same time?
Just focusing on teach for now.

Did we preserve the existing work on the Referrer API

My guess is no, but @cadecairos and @jbuck should confirm. I know we'll still have a referral system, but the API might not match.

How will we identify in the database

If that's not implemented in the storage layer right now, we can add it in. For now, all users are teach users, so we should be able to add in a system to distinguish accounts from users later on without blocking this heartbeat. @jbuck @cadecairos ?

[adamlofting]

I've just opened PR #93 to add basic pageview tracking using react-ga

I started walking through the UX. Notes to self: /login /signup /migrate /reset-password

But I can't create accounts etc yet locally to see all the places that will need instrumentation. I'm catching up with @secretrobotron later today, and will update this ticket further after that.

[adamlofting]

GA property is now setup: UA-49796218-21

[adamlofting]

Optimizely has landed and can be switched on/off with an environment variable so we're only loading the extra script while we're testing. (OPTIMIZELY_ACTIVE)

[adamlofting]

I've just updated the status of each of the checkbox items on this ticket.

We're getting close to being done, and if we keep the momentum I think this will all be ready for Friday.

[secretrobotron]

Most of this is done, and what isn't doesn't need to block. @adamlofting will move what's left into tickets and close this by tomorrow.

[adamlofting]

id.w.o will launch with better instrumentation than our existing webmaker properties. There are a few remaining tasks that won't make sense until we roll-out id to existing webmaker tools and services, and I've moved these to a backlog ticket so that we can close this.

Especially big thanks to @alicoding for implementing the GA tracking, and @jbuck for the oAuth client logging table 💫 💥 🌟 💥 💫