Since Release of Train 116, Growing Number of utm_source = (none)

[davismtl]

This might not be a bug but I want to make sure that this is the expected behavior.

We are observing an increase in utm_source = (none) at the expense of utm_source=email which is declining. We made changes to this because email was over counting so it was to be expected. However, I was expecting to see the change for registration parameters but not so much for logins.

This chart shows an increase of login with no utm source:
https://analytics.amplitude.com/mozilla-corp/chart/2ngrhz1

Confirmation that it aligns with our train 116 release:
https://analytics.amplitude.com/mozilla-corp/chart/2ngrhz1/edit/i6w4h1h

How confident do we feel in the new numbers now? Is this what we were expecting to see change? How would we explain the change in login utm params?

[philbooth]

Previously we were setting utm_source=email on every link in every email we send out. Now we set it it on none of them. So based on that, the rise in (none) seems expected to me.

Of course, whether it was the correct thing to do is another question altogether!

The only places we set utm_source=email now are on the sign-in buttons in the connect-another-device view and on the adjust app store links. I'm happy to set it in other places if it needs to be there, wherever there is. (or to revert the changes if they were the wrong thing to do)

[philbooth]

(this was the fix for mozilla/fxa-content-server#6258)

[davismtl]

Previously we were setting utm_source=email on every link in every email we send out. Now we set it it on none of them. So based on that, the rise in (none) seems expected to me.

It seems to me that in order to fix the bug we introduced with the email tracking improvements (landed last April), we decided to pull out the tracking all together. I'm not sure how we came to that conclusion.

I'm realizing that our vidyo conversation with Shane is not reflected in that bug so the conclusion and next steps don't seem well documented. However, I can see that @shane-tomlinson was on the right path when he said (in mozilla/fxa-auth-server#2505 (comment)):

This situation is a bit strange, because there are two potential flows involved. We have the flow being completed, and the potential new flow. The new flow should have a utm_source=email and we propagate that field here.

Let me recap the conversation and the content from the other bugs:

• utm params for email are good, we wanted them to see how many logins we were generating on other devices.
• however, they should not overwrite the utm params from the previously completed funnel, such as registrations.
• per the previous point, we were unfortunately changing the utm parameters for registrations after the registration because of an email click. That was an incorrect behavior. We gave the analogy of tracking an ad campaign with utm params. A click on an email should not change where a user was acquired from.
• that being said, only a login flow resulting from a click in an email confirmation should ever be tracking a utm_source=email because that is the only logical next step (other than installing the app)

The only places we set utm_source=email now are on the sign-in buttons in the connect-another-device view and on the adjust app store links. I'm happy to set it in other places if it needs to be there, wherever there is. (or to revert the changes if they were the wrong thing to do)

So, I think there is a misunderstanding here too. I asked for clarification here: mozilla/fxa-auth-server#2505 (comment)

I'm not sure I understand where in the Adjust link you are setting this and what you are replacing.

---

I would like to take a pause on this momentarily and get @irrationalagent to better document all the parameters and what values we expect to see when and why. We are starting to have regular confusion around utm properties and their proper values both (both within our team and with our reliers).

Leif, engineering does a great job working documenting request parameters here:
https://github.com/mozilla/fxa-oauth-server/blob/master/docs/api.md

We should build on top of this and add the missing utm properties and the expected values for most common use cases.

[philbooth]

I'm not sure how we came to that conclusion.

I came to that conclusion from the Vidyo conversation with you and Shane. I thought it was what we agreed on. Evidently I misunderstood.

I'm not sure I understand where in the Adjust link you are setting this and what you are replacing.

I added it to the end of the URLs in these config settings:

https://github.com/mozilla/fxa-auth-server/blob/2bfa482d45084baa429ffee59b36b734166d0cca/config/index.js#L265-L274

It is not replacing anything. It was already happening before because we were forcing utm_source=email on every link in our outgoing emails, all I did was maintain it on these particular links by setting it in config. I did that because I was asked to in the review comments for mozilla/fxa-auth-server#2505.

...better document all the parameters and what values we expect to see...

This would be helpful. In the meantime let me know if you want the recent change reverted.

[davismtl]

It is not replacing anything. It was already happening before because we were forcing utm_source=email on every link in our outgoing emails, all I did was maintain it on these particular links by setting it in config. I did that because I was asked to in the review comments for mozilla/fxa-auth-server#2505.

I'm pretty sure Adjust will just drop that param since they don't support utm param IIUC.

let me know if you want the recent change reverted.

You have a busy quarter and I've bugged you enough with this metric bug for now. I will try to sync up with Leif before asking to make any further changes.

I don't seem to have repo permissions to set this as P3 and assign Leif to it. Please do so :)

[philbooth]

Closing this in favour of mozilla/fxa-auth-server#2507, which is in the correct repo. I'll link here from there too so that the preceding discussion is not lost.