Differentiate between different services/entrypoints in flow metrics graphs

[ryanfeeley]

Now that we are showing the registrations screen at the end of first run, our failure rate is going through the roof. I would argue that simply viewing the page does not indicate intent to register. Instead we should be looking for users who at least enter a valid-looking email address into th email field.

[shane-tomlinson]

Updating the title to change iframed to firstrun.

[shane-tomlinson]

@stomlinson and @vladikoff to chat about how to tackle this.

[ckarlof]

In triage, we discussed changing our signup/signin metrics to be a more conventional funnel, e.g., capture events like:

• page loaded
• user entered email
• user entered password
• user successfully submitted form
• user verified account

I probably missed some. These are ideally tagged with entry point, user session id, and relier so we can ask questions like: "which entry points are most effective in getting users to sign up" and "which reliers are most reliable in getting users to successfully signed up".

[ckarlof]

We also discussed sending these funnel metrics into GA, because they have cool tools for visualizing.

[vladikoff]

[shane-tomlinson]

Here is what @vladikoff and I are thinking. A funnel is useful for optimizing the flow, but this is not our most pressing need on a high level dashboard. For a high level dashboard, we still want to know high level success and failure rates. We currently show a aggregated success/failure rate for all sources of traffic. The firstrun flow has such high traffic that it totally overshadows all other sources of traffic.

Until the firstrun flow was introduced, all users had to perform some action to arrive at FxA; usually via a button click. This button click is a strong indication that users were at least somewhat interested and acted as a natural filter to for our stats.

With the firstrun flow, all new users are sent to FxA; no user provided signal is given, all filters are removed.

Instead of showing a single aggregate value for success/failure, it makes more sense to break the success/failure rate out by relier (for OAuth flows) and entrypoint (for Sync). Perhaps we could show an aggregate success rate for all OAuth flows as well.

This would allow us to see how successful various services are at driving signups, and whether certain entrypoints or services provide less value than others. Separating out by relier/entrypoints would allow us to tweak the flow for each individual relier/entrypoint without affecting the stats of the others.

[vladikoff]

To fix @ryanfeeley's problem we were thinking about adding fxa.content.screen.iframe.signup with entrypoint firstrun, or just fxa.content.screen.firstrun.signup. Probably the first one because it makes more sense.

[rfk]

it makes more sense to break the success/failure rate out by relier (for OAuth flows) and entrypoint (for Sync)

👍

It would still be useful to see a high-level aggregate though. Perhaps we can start it at a slightly later point in the funnel, e.g. "user brought focus to some element in the page" rather than just "user saw the page".

[rfk]

e.g. "user brought focus to some element in the page"

@shane-tomlinson says nope; the first-run flow auto-focusses on the email field when it displays the form.

[vladikoff]

@shane-tomlinson one thing with fxa.content.screen.iframe.signup, if we add that then we would have to modify our firstrun dashboard to have a larger query (if we want to compare) signup screens based on entrypoint.

[shane-tomlinson]

firstrun dashboard to have a larger query (if we want to compare) signup screens based on entrypoint.

Is the firstrun dashboard limited to firstrun data, or does it look at all Sync data? If just the firstrun, it's entrypoint is firstrun, so shouldn't have clutter from other entrypoints?

[vladikoff]

@shane-tomlinson
example, currently we have:

{
  "viz": "timeseries",
  "requests": [
    {
      "q": "sum:fxa.content.signup.success{$env,entrypoint:uitour}.as_count()",
      "style": {
        "palette": "purple"
      },
      "type": "line"
    },
    {
      "q": "sum:fxa.content.signup.success{$env,entrypoint:firstrun}.as_count()",
      "style": {
        "palette": "orange"
      },
      "type": "line"
    }
  ]
}

we should have to change the graphs to include both:

"q": "sum:fxa.content.signup.success{$env,entrypoint:firstrun}.as_count() + sum:fxa.content.iframe.signup.success{$env,entrypoint:firstrun}.as_count()",

To include old data and new data

[ckarlof]

What are the deliverables of this work? I.e., what graphs/visualizations/reports are you going to produce?

FWIW, the ones here aren't working for me: https://metrics.services.mozilla.com/accounts-dashboard/flow/

Some of my issues:

• It's not clear if "success" includes "the user verified her email". I feel it should, or alternatively, I'd like to see a separate "verified" percentage here for each entry point.
• I feel like "success"+"failure"+"bounce" should equal 100%. It looks like it's > 100%
• The KIPs on that page suggest a story of conventional funnel events (e.g., load page, engage with page, signup success), but they're not visualized like a funnel.

[rfk]

If there are enough individual requiements/deliverables involved in the cleanup here, it may be worthwhile escalating to a "signin funnel metrics" feature in Aha for planning purposes, and breaking them up into individual issues.

https://metrics.services.mozilla.com/accounts-dashboard/flow/

I also wonder if datadog is better suited to doing the visualisations on that page, rather than having to go through the whole metrics pipeline.

[vladikoff]

We will try again to get the filter working locally, if it fails then we will proceed from there on...

[vladikoff]

Woohoo!

[rfk]

re-marking this as blocked on metrics team r?

[vladikoff]

ping needed if no activity next meeting

[vladikoff]

for next 2 weeks @philbooth ?

[philbooth]

for next 2 weeks @philbooth ?

@vladikoff, hopefully yep but I'm not promising anything. 😁

[vladikoff]

Wooo @rfk!

[rfk]

We're giving up on the approach linked above and will revisit this using new metrics infra, leaving assigned to me to come up with a next step

[rfk]

We're going to explore the next iteration of these graphs as part of mozilla/fxa-features#13 so I'm closing out the issue here.