utm_source = email for registration and login complete events

[irrationalagent]

opening this to confirm that this is logged when (and only when) a user confirms their email (both login and registration), on a device other than the one they began the signup process on.

In amplitude, only fxa_login - complete and fxa_reg - complete seem to have utm_source = email (fxa_reg - view or other earlier steps in the funnel do not). This indicates to me that utm_source is set separately for every event in the funnel, and does not necessarily reflect whatever the original starting source was. Am I right about this?

On the other hand, entrypoint seems to be consistent throughout the funnel reflecting the originating point of the flow.

@davismtl If all of the above is basically right, then we should probably be grouping our charts by entrypoint rather than utm_source if we want to know where a completed flow started out.

@mozilla/fxa-core can someone who knows the answer to this off the top of their head give us a pointer here? thanks!

[davismtl]

This indicates to me that utm_source is set separately for every event in the funnel, and does not necessarily reflect whatever the original starting source was. Am I right about this?

This is not quite accurate. UTM params should be about the top of funnel. We want to know where people started. After all, UTM params are meant for tracking advertising channels which are top of funnel.

Even if your statement was correct, it seems to be that it would be impossible that the previous step to a registration was clicking on a link from one of our confirmation emails:

https://analytics.amplitude.com/org/31251/chart/oirl9vh/edit/t65fjat

To go through that email, you would have had to be registered. Therefore, you should not be in the middle of registering.

That aside, we should be more descriptive with utm_source=email. Email is a medium not a source. It leaves room for too much interpretation.

A login_complete on iOS could have for example the following utm params:
utm_source=login_confirmation(_email)
utm_medium=email

or

utm_source=reg_confirmation(_email)
utm_medium=email

and does not necessarily reflect whatever the original starting source was. Am I right about this?

In this particular case, it also makes sense for the UTM params to be different then everything before the email because in theory, the user is starting a new funnel so it would not be inaccurate to call out from which email the user came.

[vladikoff]

@davismtl what is the action item here?

[davismtl]

The scope of this problem seems to be a bit bigger in light of the conversation.

1. (initial reason for bug) Figure out why we see registrations from utm_source=email . This is normally impossible because if someone is receiving our emails, they are already registered. I suspect an engineer needs to answer this.
2. @irrationalagent please work with me so we can better define best practice around utm params (and entrypoint). When do we use which one? Are there some that can't change? etc...
3. Make appropriate changes to utm parameters so that traffic from our emails is better understood.
4. @irrationalagent Let's add our utm and entrypoint best practices (following Add browserid certifier to project #2) to our public documentation so that we can more easily share it with other teams. In an other bug, we see that the onboarding team made mistakes in about:welcome. This underlines the need for a clearer best practice.

[shane-tomlinson]

When @jpetto asked how utm_ and entrypoint are used in the context of the mozilla.org iframeless integration (not about:welcome), I responded with:

We only really have two general rules of thumb. entrypoint is used to tag a given "page type" and remains the same across all pages of the class, e.g., no matter URL of the firstrun page, entrypoint is always firstrun. The utm_* params vary more often to allow for fine grained metrics, following the semantics provided by Google.

[irrationalagent]

@mozilla/fxa-core can an engineer explain the circumstances that lead to utm_source = email for amplitude fxa_reg - complete events?

https://analytics.amplitude.com/org/31251/chart/oirl9vh/edit/t65fjat

It was my understanding that this was related to the confirmation email being clicked from another device, but according to Alex's comment above, this should not be possible.

I'll start on an audit of our utm param and entrypoint values, so that we can get on the same page about how these should be set.

[vladikoff]

Figure out why we see registrations from utm_source=email

All emails have UTM links generated in them:
https://github.com/mozilla/fxa-auth-server/blob/master/lib/senders/email.js#L1091-L1092

If I click "Verify" in any of those emails and get to the content server my utm params are 'email'. Therefore I will be counted as a registration from that new open tab. I hope that explains the metrics results

[davismtl]

If I click "Verify" in any of those emails and get to the content server my utm params are 'email'. Therefore I will be counted as a registration from that new open tab. I hope that explains the metrics results

I understand but that isn't right. Under that logic, all registrations would look like utm_source=email since they all have to verify in an email at registration and will all have a tab open.

This overwrites the true origin of the registration.

[vladikoff]

This overwrites the true origin of the registration.

That is true I think!

[shane-tomlinson]

Under that logic, all registrations would look like utm_source=email since they all have to verify in an email at registration and will all have a tab open.

From what I'm reading, it sounds like any user that verifies on a 2nd device has their utm_source updated to email? If so, that seems like a bug. It seems like a better way of handling it would be to check if a utm_source is already associated with the registration, and if so do not update.

[shane-tomlinson]

It seems like a better way of handling it would be to check if a utm_source is already associated with the registration, and if so do not update.

There is a caveat, I don't know if what I stated is possible.