Implementing A/B test on the domains step.

[donlair]

Proposed Changes

• This PR implements an A/B test for the CTA and body copy in the Skip section of the Domain step. Additional context for this test is available at p58i-cJS-p2#comment-54846

Here's a screenshot of the page (with a green arrow pointing to the text that will be tested):

Control body copy:

Use the search tool on this page to find a domain you love, then select any paid annual plan.

We’ll pay the first year’s domain registration fees for you, simple as that!

Variant body copy:

You can claim your free custom domain later if you aren’t ready yet.

Control CTA: Choose my domain later
Variant CTA: View plans

The test is implemented using Explat with the test name domain_step_cta_copy_test

Testing Instructions

• Checkout this PR and start Calypso locally.
• Navigate to the Explat page for test name domain_step_cta_copy_test.
• Assign yourself to the treatment variation (click link at the top of the Bookmarklet modal and assign your main username).
• If you have public-api sandboxed, ensure it's up-to-date.
• Navigate to startup in Calypso and confirm that the CTA says Choose my domain later.
• Assign yourself to the control variation and confirm that the CTA says View plans.
• Click the link and confirm that you're brought to the plans page.

[github-actions]

Calypso Live (direct link)

https://calypso.live?image=registry.a8c.com/calypso/app:build-39133

Jetpack Cloud live (direct link)

https://calypso.live?image=registry.a8c.com/calypso/app:build-39133&env=jetpack

[matticbot]

Here is how your PR affects size of JS and CSS bundles shipped to the user's browser:

App Entrypoints (~33 bytes added 📈 [gzipped])

name                   parsed_size           gzip_size
entry-main                   +70 B  (+0.0%)      +33 B  (+0.0%)
entry-stepper                +43 B  (+0.0%)      +26 B  (+0.0%)
entry-login                  +43 B  (+0.0%)      +26 B  (+0.0%)
entry-gutenboarding          +43 B  (+0.0%)      +26 B  (+0.0%)
entry-domains-landing        +43 B  (+0.0%)      +26 B  (+0.0%)
entry-browsehappy            +43 B  (+0.0%)      +26 B  (+0.1%)

Common code that is always downloaded and parsed every time the app is loaded, no matter which route is used.

Sections (~382 bytes added 📈 [gzipped])

name                   parsed_size           gzip_size
signup                      +291 B  (+0.1%)     +349 B  (+0.4%)
jetpack-connect             +198 B  (+0.0%)      +22 B  (+0.0%)
accept-invite               +176 B  (+0.0%)      +13 B  (+0.0%)
plans                        +22 B  (+0.0%)       +9 B  (+0.0%)
pages                        +22 B  (+0.0%)      +14 B  (+0.0%)
jetpack-cloud-pricing        +22 B  (+0.0%)       +9 B  (+0.0%)
checkout                     +22 B  (+0.0%)      +10 B  (+0.0%)

Sections contain code specific for a given set of routes. Is downloaded and parsed only when a particular route is navigated to.

Async-loaded Components (~397 bytes added 📈 [gzipped])

name                             parsed_size           gzip_size
async-load-signup-steps-domains      +7539 B  (+1.8%)    +2830 B  (+2.4%)
async-load-signup-steps-plans         +115 B  (+0.0%)     +298 B  (+0.3%)

React components that are loaded lazily, when a certain part of UI is displayed for the first time.

Legend

What is parsed and gzip size?

Parsed Size: Uncompressed size of the JS and CSS files. This much code needs to be parsed and stored in memory.
Gzip Size: Compressed size of the JS and CSS files. This much data needs to be downloaded over network.

Generated by performance advisor bot at iscalypsofastyet.com.

[DavidRothstein]

Looking at this more closely, I think that in order to make sense, it will actually need to change more than just the CTA?

That's because the current text is all about telling people to search for a domain on the current page... but then "View plans" does the exact opposite and it isn't clear that it will.

So instead of this:

Would it be better to go back and test the original text (that existed before #62243)?:

Or alternatively, some sensible combination of them? (Although it may be best to start by just testing exactly what was there before.)

[donlair]

Would it be better to go back and test the original text (that existed before #62243)?:

Good catch @DavidRothstein, I updated the PR to change the body copy as well.

[DavidRothstein]

I'm not too familiar with the A/B test code myself, but overall this looks good to me now - thanks!

[gmovr]

Tested and works well. Code LGTM! 🚢

The new copy isn't translated, but I see in the experiment UI that we're just running this for EN locales, so we should be good (just wanted to mention it, in case we roll it out to more locales).

[a8ci18n]

This Pull Request is now available for translation here: https://translate.wordpress.com/deliverables/7464967

Thank you @donlair for including a screenshot in the description! This is really helpful for our translators.

[razvanpapadopol]

Hi, @donlair 👋🏼
I was reviewing this PR when it got merged but I'll leave a comment since there are two aspects worth discussing:

Locale targeting

Regarding what @gmovr shared, it's a possibility that it may affect negatively Treatment users. Even if the experiment, as set in Abacus, is going to run only for EN locale, a user that is assigned to Treatment may be accessing this screen also with another locale by switching to their native language or by using a different browser.

I don't think this is hugely worrying (cc: @jessie-ross for a second opinion) but I'll mention this also in the next point.

Handling Experiment data loading

When first landing on this screen, before the experiment data is retrieved, all users will see for a second or more (depending on the internet connection) the subtitles and CTA of the Control experience. So it will cause a slight decrease in UX for users assigned to Treatment as they will experience some flickering. I've suggested a way to mitigate this in #66316 where we also use the locale to limit the placeholder usage only for users targeted by this experiment (EN locale).

For a quicker loading, I remember using client/layout/index.jsx to fire the loadExperimentAssignment call in order to preload the experiment data for the time it will be evaluated. Usually, for a new user, that means the experiment call is done on signup/login step of the flow.

[jessie-ross]

I don't think this is hugely worrying (cc: @jessie-ross for a second opinion) but I'll mention this also in the next point.

Yeah this is a very good point and one we haven't looked into much. It's hard to say as we don't have stats on "segment crossovers" like this. I feel like it is safe to assume in the meantime that such instances are sufficiently small...

[a8ci18n]

Translation for this Pull Request has now been finished.