Create an experiment for the download funnel

[bobsilverberg]

Depends on mozilla/bedrock#10207

Implement an experiment to test removing a step from the download funnel for non-Firefox users. Users in the test cohort would be served a link which points to a special page on mozilla.org which starts the download automatically, while those on the control cohort would be served a link which points to the /new page.

Because the special page, which will be a version of the /thanks page, is unable to handle users with IE 9 and below, we need to be able to exclude those users from the experiment. Ideally we would exclude them entirely, meaning they wouldn't be placed into either cohort and wouldn't affect the data we are collecting, but considering the small number of these users, it might be acceptable to simply force them into the control cohort. According to GA, we have had 40 users with IE 9 or below over the past 2 weeks.

For QA:

The experiment should place users in one of two cohorts, and depending on which cohort you are in, you will see a different URL for downloading Firefox on the site. The new URL is https://www.mozilla.org/firefox/download/thanks/, and when that URL is used there should be an additional query param of source=amo added to it. Note that this new URL will not actually work for downloading Firefox yet. It will only start working when mozilla/bedrock#10207 lands. The experiment should be active on dev only right now.

[ioanarusiczki]

@bobsilverberg

I tested on Edge/Chrome and so far here are my results with AMO dev on Win10:

• I see a new cookie set on AMO dev : %7B%2220210531_download_funnel_experiment%22%3A%22new-link%22%7D
• when the new link is set there's a source=amo parameter added

• on download URL https://www.mozilla.org/firefox/download/thanks/, there's a blocked:csp error I could reproduce with Edge and Chrome

this is the network:

and the console:

• I tested and reproduced with VM - Sandbox this: when I have the new-link experiment set and click on a download button from AMO for the first time, I land on https://www.mozilla.org/en-US/firefox/download/new/. I go back and click again a download button from AMO and now I am redirected correctly to https://www.mozilla.org/en-US/firefox/download/thanks/

• I did not test this on Android yet, but it should be available there too.
• the download is triggered - installs can be made, I'll test more around the way they're triggered tomorrow.

[bobsilverberg]

@bobsilverberg

I tested on Edge/Chrome and so far here are my results with AMO dev on Win10:

• I see a new cookie set on AMO dev : %7B%2220210531_download_funnel_experiment%22%3A%22new-link%22%7D

• when the new link is set there's a source=amo parameter added

Those are good.

• on download URL mozilla.org/firefox/download/thanks, there's a blocked:csp error I could reproduce with Edge and Chrome

This is a CSP error you are seeing on mozilla.org, correct (i.e. not on AMO)? If that is the case this is something I guess we'd need to take up with them, but note that the page we are planning to direct to hasn't been fully implemented for us yet, so perhaps this will go away when it has been.

• I tested and reproduced with VM - Sandbox this: when I have the new-link experiment set and click on a download button from AMO for the first time, I land on mozilla.org/en-US/firefox/download/new. I go back and click again a download button from AMO and now I am redirected correctly to mozilla.org/en-US/firefox/download/thanks

This sounds like a bug. I have done some more work on the experiment infrastructure in mozilla/addons-frontend#10665, during which I noticed something similar, so perhaps this will be fixed via that PR. Can you open a separate issue about this and we can track it there?

• I did not test this on Android yet, but it should be available there too.

• the download is triggered - installs can be made, I'll test more around the way they're triggered tomorrow.

As discussed in the "For QA:" section above, the code for this new /thanks/ page has not been completed yet, so while it probably will work for downloading Firefox, there will be other issues with is (such as not including the attribution code). I think for now all we need verified is that the experiment works and does direct users to different URLs based on which cohort they are in, and that we don't necessarily need to verify the download triggering part on mozilla.org until they have completed their work to support our use case.