Closeout for sign-in confirmation #193
Conversation
|
|
||
| <img src="sync-sign-in-success-rate.png" height="300"> | ||
|
|
||
| This meets our original goal of not effecting this metric. |
rfk
Sep 23, 2016
Member
nit: "affecting"
nit: "affecting"
|
What would be your overall conclusion? |
| Pre sign-in confirmation metrics show that we had between a 42% to 52% | ||
| sync sign-in success rate. From [first dashboard](https://app.datadoghq.com/dash/163668/fxa-content---signin-confirmation), with sign-in confirmation, we have maintained this sign-in rate. | ||
| Pre sign-in confirmation metrics show that we had | ||
| between a 42% to 52% sync sign-in success rate. |
davismtl
Sep 23, 2016
Contributor
I have trouble with this metric when I look at the details in datadog.
(fxa.content.signin.signin.success + fxa.content.force_auth.signin.success + fxa.content.complete_signin.verification.success) / (fxa.content.signin.submit + fxa.content.force_auth.submit) * 100
Why are there 3 different login success events?
Why are there 2 different form submit events?
I've tried to piece things together (by reading this) but it's a bit unclear to me and some events are missing I think.
I have trouble with this metric when I look at the details in datadog.
(fxa.content.signin.signin.success + fxa.content.force_auth.signin.success + fxa.content.complete_signin.verification.success) / (fxa.content.signin.submit + fxa.content.force_auth.submit) * 100
Why are there 3 different login success events?
Why are there 2 different form submit events?
I've tried to piece things together (by reading this) but it's a bit unclear to me and some events are missing I think.
shane-tomlinson
Sep 26, 2016
Member
Why are there 3 different login success events?
Why are there 2 different form submit events?
This is the most common question we have about the metrics and why I proposed mozilla/fxa-content-server#4163.
Most reported event names contain a built in screen name. This is so we know on which screen an event originates. Logging success without any additional context is a pointless metric - "what kind of success?" is the obvious question that follows. Way back when, there was only 1 way to sign in, from the /signin screen. At that time it was sufficient to have fxa.content.signin.success, we knew what signin.success meant and where it was coming from. Users can now sign in from 6 different endpoints: /signin, /force_auth, /signup, /oauth/signin, /oauth/force_auth, and /oauth/signup. When we started to add endpoints, we still wanted to know from which screen the user was successfully signing in, so we "extended" the event name - fxa.content.<screen_name>.signin.success. It has become unruly because there is no single fxa.content.signin.success anymore.
Why are there 3 different login success events?
Why are there 2 different form submit events?
This is the most common question we have about the metrics and why I proposed mozilla/fxa-content-server#4163.
Most reported event names contain a built in screen name. This is so we know on which screen an event originates. Logging success without any additional context is a pointless metric - "what kind of success?" is the obvious question that follows. Way back when, there was only 1 way to sign in, from the /signin screen. At that time it was sufficient to have fxa.content.signin.success, we knew what signin.success meant and where it was coming from. Users can now sign in from 6 different endpoints: /signin, /force_auth, /signup, /oauth/signin, /oauth/force_auth, and /oauth/signup. When we started to add endpoints, we still wanted to know from which screen the user was successfully signing in, so we "extended" the event name - fxa.content.<screen_name>.signin.success. It has become unruly because there is no single fxa.content.signin.success anymore.
shane-tomlinson
Sep 26, 2016
•
Member
I forgot, the signin event is further complicated by signin confirmation and the coming signin unblock, and even OAuth permissions which we don't currently include in our metrics.
Figuring out what signin.success means is not an easy task. Does it mean the user has fully completed the signin flow, post email verification? With signin confirmation, we can't measure that using only client side metrics because the client doesn't always know when the user has completed the flow since users sometimes close FxA before verifying their email.
I forgot, the signin event is further complicated by signin confirmation and the coming signin unblock, and even OAuth permissions which we don't currently include in our metrics.
Figuring out what signin.success means is not an easy task. Does it mean the user has fully completed the signin flow, post email verification? With signin confirmation, we can't measure that using only client side metrics because the client doesn't always know when the user has completed the flow since users sometimes close FxA before verifying their email.
rfk
Sep 26, 2016
Member
+1 to what @shane-tomlinson said; my high-level thinking is basically that we want an explicit "you completed what you were trying to do" event that's emitting at the very last stage of the flow. For sync signins that's when the browser actually succeeds in connecting to sync. For oauth signins that's when the relier successfully obtains their access token. Neither of those can be measured using only client-side metrics.
IOW, bring on the flow.completed flow event for future explorations of this type :-)
+1 to what @shane-tomlinson said; my high-level thinking is basically that we want an explicit "you completed what you were trying to do" event that's emitting at the very last stage of the flow. For sync signins that's when the browser actually succeeds in connecting to sync. For oauth signins that's when the relier successfully obtains their access token. Neither of those can be measured using only client-side metrics.
IOW, bring on the flow.completed flow event for future explorations of this type :-)
|
|
||
| <img src="sign-in-success-time.png" height="300"> | ||
|
|
||
| This chart shows that most users complete sign-in |
davismtl
Sep 23, 2016
Contributor
How does it help us to know that confirmation happens usually within an hour?
How does it help us to know that confirmation happens usually within an hour?
vbudhram
Sep 27, 2016
Author
Contributor
I think this helps us eliminate some reasons why the server-side metrics and client-side metrics differed on sign-in confirmation.
I think this helps us eliminate some reasons why the server-side metrics and client-side metrics differed on sign-in confirmation.
| in these metrics is that one method might not be emitting | ||
| or emitting twice, for different scenarios. The true | ||
| sign-in confirmation success rate could possibly be a | ||
| mix of these metrics. |
davismtl
Sep 23, 2016
Contributor
Do we have telemetry that would help us answer this? Were we tracking all the way back then?
Do we have telemetry that would help us answer this? Were we tracking all the way back then?
+1, I think this is worth trying to summarize explicitly. From my perspective the feature seems worthwhile on balance, but we definitely pay a tax for the increased security, so follow-up work is to try different ways of avoiding sending the email when it's safe to do so. |
|
Looks good to me. Only 2 little changes but nothing serious |
| email. Sign-in confirmation helps to provide these users | ||
| with some protection on their accounts. | ||
|
|
||
| <img src="sign-in-change-password.png" height="300"> |
davismtl
Sep 30, 2016
Contributor
This image isn't loading for me. :-/
This image isn't loading for me. :-/
vbudhram
Oct 3, 2016
Author
Contributor
|
|
||
| <img src="sign-in-change-password.png" height="300"> | ||
|
|
||
| Overall, the sign-in confirmation feature has provided |
davismtl
Sep 30, 2016
Contributor
Prob worth making everything past this it's own section so that it pops. Conclusion and Next Steps
Prob worth making everything past this it's own section so that it pops. Conclusion and Next Steps
This PR adds result and dashboards from sign-in confirmation.
Ref: mozilla/fxa-features#3