Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable domain step design update test for test environments #39276

Merged
merged 1 commit into from Feb 11, 2020

Conversation

niranjan-uma-shankar
Copy link
Contributor

@niranjan-uma-shankar niranjan-uma-shankar commented Feb 5, 2020

Changes proposed in this Pull Request

  • This PR creates a new A/B test for the domain copy updates changes as described in pbAok1-7N-p2.
  • There's nothing more happening here other than enabling the config flag and the A/B test.

What next

Subsequent UI updates will be built on top of this. Check below to see the referenced PRs to see all the smaller tasks that build each piece of the UI update. When all the smaller pieces are finished, then a new PR that removes the feature flag will be deployed.

Testing instructions

  • This PR introduces no UI changes, so everything should work just as before. Only change is that in dev and wpcalypso environments, user will get assigned to a new AB test when they reach the domain step in signup flow.

In dev or wpcalypso environments

Use http://calypso.localhost:3000/ URL for dev environment or https://calypso.live/?branch=add/domain-step-signup-v2-abtest for wpcalypso environment.

  1. Go through the signup flow by navigating to http://calypso.localhost:3000/start or https://calypso.live/start?branch=add/domain-step-signup-v2-abtest. Once you reach the domain step, verify that you get assigned to two tests - domainStepCopyUpdates and domainStepDesignUpdates. You can verify this by typing localStorage.ABTests; in your browser console.
  2. Verify the following:
    • If in the control group of domainStepCopyUpdates, no change in behaviour i.e you will see the original domain step UI.
    • If in the variant of domainStepCopyUpdates, the domain step UI will depend on the other test:
  3. Verify that you are able to complete signup.

In production environment

There should be no change in behaviour since we are hiding the new AB test behind a feature flag.
To simulate the production environment, we will turn off the domain-step-design-update-v2 feature flag in the URL.

  1. Go through the signup flow by navigating to http://calypso.localhost:3000/start?flags=-domain-step-design-update-v2 or if using calypso.live URL, append ?flags=-domain-step-design-update-v2 to the hashed calypso.live link e.g: https://hash-ed3926da93141d2ed4ab69328cdf577f828561d8.calypso.live/start?flags=-domain-step-design-update-v2
  2. In the domain step, you should be assigned to only domainStepCopyUpdates test and should not be assigned to domainStepDesignUpdates.
  3. Verify that you can complete signup.

Fixes 192-gh-martech

@niranjan-uma-shankar niranjan-uma-shankar requested a review from a team as a code owner February 5, 2020 11:50
@niranjan-uma-shankar niranjan-uma-shankar self-assigned this Feb 5, 2020
@matticbot
Copy link
Contributor

@niranjan-uma-shankar niranjan-uma-shankar added [Status] In Progress [Status] Needs Review The PR is ready for review. This also triggers e2e canary tests and wp-desktop tests automatically. [Feature] Signup & Account Creation All screens and flows for making a new WordPress.com account. ABTest [Feature Group] Emails & Domains Features related to email integrations and domain management. labels Feb 5, 2020
@matticbot
Copy link
Contributor

Here is how your PR affects size of JS and CSS bundles shipped to the user's browser:

App Entrypoints (~110 bytes added 📈 [gzipped])

name                   parsed_size           gzip_size
entry-main                  +144 B  (+0.0%)      +26 B  (+0.0%)
entry-login                 +144 B  (+0.0%)      +21 B  (+0.0%)
entry-jetpack-cloud         +144 B  (+0.0%)      +20 B  (+0.0%)
entry-gutenboarding         +144 B  (+0.0%)      +23 B  (+0.0%)
entry-domains-landing       +144 B  (+0.0%)      +20 B  (+0.0%)

Common code that is always downloaded and parsed every time the app is loaded, no matter which route is used.

Async-loaded Components (~60 bytes added 📈 [gzipped])

name                             parsed_size           gzip_size
async-load-signup-steps-domains       +186 B  (+0.1%)      +60 B  (+0.1%)

React components that are loaded lazily, when a certain part of UI is displayed for the first time.

Legend

What is parsed and gzip size?

Parsed Size: Uncompressed size of the JS and CSS files. This much code needs to be parsed and stored in memory.
Gzip Size: Compressed size of the JS and CSS files. This much data needs to be downloaded over network.

Generated by performance advisor bot at iscalypsofastyet.com.

Copy link
Contributor

@southp southp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works as expected. LGTM 🚢
However, I didn't inspect localStorage.ABTests to see my assigned variation. It didn't work for me, which I think expected because the store package doesn't always store values there. Instead, I manually logged abtest( ... ) to see the assigned variations.

client/signup/steps/domains/index.jsx Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ABTest [Feature Group] Emails & Domains Features related to email integrations and domain management. [Feature] Signup & Account Creation All screens and flows for making a new WordPress.com account.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants