Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Wrapping up the Delete/Block experiment #864
The experiment #732 has been running for a while, and collecting enough data from the user's interaction with the new context menu.
We should be able to come to the conclusion with that data, and figure out the following actions if necessary accordingly.
can we get all our experiments documented centrally. Here's how the growth team does it: (internal link)
Just conducted some preliminary analysis on this experiment, we'd better make a formal report later based on this if necessary.
To study that whether or not the addon user knows the difference between "dismiss" and "delete" in the context menu.
The whole population has been divided into two equal size groups, each group uses a slightly different context menu, i.e., the control group uses a menu with option "dismiss", followed by the "delete". While the experiment group uses the menu with the reverse options.
In each group, we've collected the total number of unique clients who have clicked on the "delete" and "dismiss" group by date. The null hypothesis is "The total numbers of unique users who have clicked on the "delete/dismiss" are the same in those two groups".
As for "delete", the p-value of Welch's t-test is 0.832. And for "dismiss", the p-value of Welch's t-test is 0.207. Neither of them is able to reject the null hypothesis in this experiment, which means that the users knew what they wanted to do, they didn't just happen to click on the first option of the menu.
Delete vs. Dismiss
Apparently, there are more users who have clicked on the option "delete" than that of "dismiss", the p-value is 0.003 < α(5%). Note that 781 ouf of 3093 (25.2%) users have clicked on both options in the experiment, hence, the "dismiss" option is still useful for some users.
We've applied both the independent two-sample t-test and the Welch's t-test on the experiment data. The p-values of those two tests are almost identical, that indicates the variances of those two populations are almost the same.
Nice work @ncloudioj! I reviewed your analysis and it looks good! So I'd say there are at least two interesting conclusions here: