Projects: experiment name chevron-publish -> chevron-publish-v2

[Erin007]

There were a couple of major flaws with my initial set up for the A/B test of publishing in different ways from the PersonalProjectsTable.

1.) I was logging a 'page-load' every time the table (re)rendered rather than only when the projects page was loaded. Fixed in #24527.

2.) I timed the set up of my user based experiment on production too late based on my other changes so there was a duration where everyone was only seeing one variation.

To collect accurate data, I've updated the experiment name from chevron-publish to chevron-publish-v2 in this PR, and set up new experiments on production and staging and stopped the previous version.

staging

set up new chevron-publish-v2

stop chevron-publish

production

set up new chevron-publish-v2

stop chevron-publish

Note, there is potential for slight user bad with this change. Someone previously in one variation of the study might be randomized into the other, which could be confusing. Hopefully, publishing is straightforward enough via either UI that it won't be too disruptive - and we're about to find out which method is more discoverable!

[davidsbailey]

Hi Erin, apologies for suggesting an approach which ended up being this complicated. My vision was for you to only to have to update the strings we were using to log things, but I forgot that the study name was tied to the experiment name and should have mentioned that goal more generally.

If you wanted to eliminate the user-bad you mentioned, you could append -v2 to the two study_groups instead of appending it to the study.

However, since you've already done the work of setting up the new experiment, I think its fine to proceed with what you have here. We can just remember to find a way to reset the data less disruptively next time.

[Erin007]

I also think the user bad isn’t really that bad because switching variants that the user sees is something we’ve planned for the end of the experiment anyway. Those in the “losing” variant will be switched to the “winning” one; an unlucky few might just be flipped between versions an extra time.

Thanks for the clarification, Dave! I should’ve asked more questions before I proceeded.