Wrapping up the Delete/Block experiment #864

Closed
ncloudioj opened this Issue Jun 20, 2016 · 5 comments

Comments

Projects
None yet
5 participants
@ncloudioj
Contributor

ncloudioj commented Jun 20, 2016

The experiment #732 has been running for a while, and collecting enough data from the user's interaction with the new context menu.

We should be able to come to the conclusion with that data, and figure out the following actions if necessary accordingly.

What do you think? @tspurway @k88hudson @emtwo

@ncloudioj ncloudioj added this to the Columbia (Jun 25) milestone Jun 20, 2016

@ncloudioj ncloudioj self-assigned this Jun 20, 2016

@edwinwongsf

This comment has been minimized.

Show comment
Hide comment
@edwinwongsf

edwinwongsf Jun 21, 2016

can we get all our experiments documented centrally. Here's how the growth team does it: (internal link)
https://mana.mozilla.org/wiki/display/FIREFOX/Detailed+Experiment+Reports+of+Firefox+Growth+Team

can we get all our experiments documented centrally. Here's how the growth team does it: (internal link)
https://mana.mozilla.org/wiki/display/FIREFOX/Detailed+Experiment+Reports+of+Firefox+Growth+Team

@ncloudioj

This comment has been minimized.

Show comment
Hide comment
@ncloudioj

ncloudioj Jun 21, 2016

Contributor

Just conducted some preliminary analysis on this experiment, we'd better make a formal report later based on this if necessary.

Experiment Goal

To study that whether or not the addon user knows the difference between "dismiss" and "delete" in the context menu.

Experiment Design

The whole population has been divided into two equal size groups, each group uses a slightly different context menu, i.e., the control group uses a menu with option "dismiss", followed by the "delete". While the experiment group uses the menu with the reverse options.

Statistical Analysis

In each group, we've collected the total number of unique clients who have clicked on the "delete" and "dismiss" group by date. The null hypothesis is "The total numbers of unique users who have clicked on the "delete/dismiss" are the same in those two groups".

Delete

group total # of unique users group by date (2016-06-12 to 2016-06-21) total # of unique users
control 142 ,174 ,145 ,204 ,185 ,187 ,161 ,139 ,180 ,113 1126
exp 140 ,160 ,189 ,202 ,220 ,167 ,148 ,133 ,182 ,118 1062

Dismiss

group total # of unique users group by date (2016-06-12 to 2016-06-21) total # of unique users
control 90 ,121 ,142 ,179 ,174 ,138 ,100 , 97 ,142 ,113 905
exp 90 ,116 , 97 ,147 ,159 ,129 ,116 , 85, 111, 76 781

As for "delete", the p-value of Welch's t-test is 0.832. And for "dismiss", the p-value of Welch's t-test is 0.207. Neither of them is able to reject the null hypothesis in this experiment, which means that the users knew what they wanted to do, they didn't just happen to click on the first option of the menu.

Delete vs. Dismiss

Event total # of unique users group by date (2016-06-12 to 2016-06-21) Total of unique users
delete 282, 334, 334, 406, 405, 354, 309, 272, 362, 231 2188
dismiss 180, 237, 239, 326, 333, 267, 216, 182, 253, 189 1686

Apparently, there are more users who have clicked on the option "delete" than that of "dismiss", the p-value is 0.003 < α(5%). Note that 781 ouf of 3093 (25.2%) users have clicked on both options in the experiment, hence, the "dismiss" option is still useful for some users.

Note

We've applied both the independent two-sample t-test and the Welch's t-test on the experiment data. The p-values of those two tests are almost identical, that indicates the variances of those two populations are almost the same.

Contributor

ncloudioj commented Jun 21, 2016

Just conducted some preliminary analysis on this experiment, we'd better make a formal report later based on this if necessary.

Experiment Goal

To study that whether or not the addon user knows the difference between "dismiss" and "delete" in the context menu.

Experiment Design

The whole population has been divided into two equal size groups, each group uses a slightly different context menu, i.e., the control group uses a menu with option "dismiss", followed by the "delete". While the experiment group uses the menu with the reverse options.

Statistical Analysis

In each group, we've collected the total number of unique clients who have clicked on the "delete" and "dismiss" group by date. The null hypothesis is "The total numbers of unique users who have clicked on the "delete/dismiss" are the same in those two groups".

Delete

group total # of unique users group by date (2016-06-12 to 2016-06-21) total # of unique users
control 142 ,174 ,145 ,204 ,185 ,187 ,161 ,139 ,180 ,113 1126
exp 140 ,160 ,189 ,202 ,220 ,167 ,148 ,133 ,182 ,118 1062

Dismiss

group total # of unique users group by date (2016-06-12 to 2016-06-21) total # of unique users
control 90 ,121 ,142 ,179 ,174 ,138 ,100 , 97 ,142 ,113 905
exp 90 ,116 , 97 ,147 ,159 ,129 ,116 , 85, 111, 76 781

As for "delete", the p-value of Welch's t-test is 0.832. And for "dismiss", the p-value of Welch's t-test is 0.207. Neither of them is able to reject the null hypothesis in this experiment, which means that the users knew what they wanted to do, they didn't just happen to click on the first option of the menu.

Delete vs. Dismiss

Event total # of unique users group by date (2016-06-12 to 2016-06-21) Total of unique users
delete 282, 334, 334, 406, 405, 354, 309, 272, 362, 231 2188
dismiss 180, 237, 239, 326, 333, 267, 216, 182, 253, 189 1686

Apparently, there are more users who have clicked on the option "delete" than that of "dismiss", the p-value is 0.003 < α(5%). Note that 781 ouf of 3093 (25.2%) users have clicked on both options in the experiment, hence, the "dismiss" option is still useful for some users.

Note

We've applied both the independent two-sample t-test and the Welch's t-test on the experiment data. The p-values of those two tests are almost identical, that indicates the variances of those two populations are almost the same.

@ncloudioj

This comment has been minimized.

Show comment
Hide comment
@ncloudioj

ncloudioj Jun 21, 2016

Contributor

@edmoz Good point. As I mentioned above, we should have a home for all the experiments' details (design, data, analysis, conclusion etc.). Wherever it is, everyone should be able to access it easily.

Contributor

ncloudioj commented Jun 21, 2016

@edmoz Good point. As I mentioned above, we should have a home for all the experiments' details (design, data, analysis, conclusion etc.). Wherever it is, everyone should be able to access it easily.

@emtwo

This comment has been minimized.

Show comment
Hide comment
@emtwo

emtwo Jun 22, 2016

Collaborator

Nice work @ncloudioj! I reviewed your analysis and it looks good! So I'd say there are at least two interesting conclusions here:

  1. Users know whether they want to delete or dismiss; the order of the options does not affect them
  2. Both options are used and worth keeping
Collaborator

emtwo commented Jun 22, 2016

Nice work @ncloudioj! I reviewed your analysis and it looks good! So I'd say there are at least two interesting conclusions here:

  1. Users know whether they want to delete or dismiss; the order of the options does not affect them
  2. Both options are used and worth keeping
@k88hudson

This comment has been minimized.

Show comment
Hide comment
@k88hudson

k88hudson Jun 23, 2016

Member

Awesome work!!

Member

k88hudson commented Jun 23, 2016

Awesome work!!

@oyiptong oyiptong modified the milestone: Columbia (Jun 25) Jun 27, 2016

@oyiptong oyiptong closed this Jun 27, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment