Native Jetpack Installation: Apply A/B test

[selanthiraiyan]

Closes: #8166

Description

This PR adds a new A/B test experiment for simplified login to enable native Jetpack installation flow. Experiment details:

• Added new A/B test experiment.
• Replaced the temporary feature flag with A/B test.

Testing instructions

Prerequisite: Make sure that you have a test store that is not associated with your WP.com account or doesn't have Jetpack.

control variation

• Log out of the app or skip login onboarding if needed.
• On the prologue screen, select "Log In" or "Continue with WordPress.com" based on the A/B test variant you get.
• Log in with your WordPress.com account.
• If your account doesn't have any associated site, notice that the empty store picker is displayed with the matching design as on Figma.
• Select Add a Store > Connect an existing site.
• Enter the address of your test store and tap Continue.
• You will see the existing Jetpack error screen.
• You can install and setup Jetpack in a webview from this screen.

Testing the treatment variation

We will have to sandbox and assign a particular variation to the device to test the variations.

Prerequisites:

• Please follow the setup in PCYsg-Fq7-p2#assignments-api for proxied sandboxed API requests: connect to sandbox via ssh, and point WP.com API to the sandbox IP address (ifconfig) in the local matchine's /etc/hosts
• In Xcode, set a breakpoint at ExPlatService L47 where anon_id is set in the URL request
• Make an API request PATCH /wpcom/v2/experiments/0.1.0/assignments with the following body and make sure it's successful:

{
    "variations": {
        "woocommerceios_login_jetpack_setup_flow": "treatment"
    },
    "anon_id": "anon_id_from_the_previous_step",
    "username_override": ""
}

• Launch the app, and let it sit for a bit
• Close and relaunch the app
• Log out of the app or skip login onboarding if needed.
• On the prologue screen, select "Log In" or "Continue with WordPress.com" based on the A/B test variant you get.
• Log in with your WordPress.com account.
• If your account doesn't have any associated site, notice that the empty store picker is displayed with the matching design as on Figma.
• Select Add a Store > Connect an existing site.
• Enter the address of your test store and tap Continue.
• You will see the new Jetpack error screen. (Screenshot below)
• You can install and setup Jetpack in new native flow starting from this screen.

Screenshots

Control	Treatment

---

• I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

[wpmobilebot]

You can test the changes from this Pull Request by:

• Clicking here or scanning the QR code below to access App Center
• Then installing the build number pr8170-f61ea32 on your iPhone

If you need access to App Center, please ask a maintainer to add you.

[jaclync]

@selanthiraiyan I was testing the treatment case, but I couldn't get the variation after following the setup. The API request to set the assignment manually worked, as I saw the assignments with woocommerceios_login_jetpack_setup_flow: treatment initially for the anon ID. However, when logging in, the app calls ABTest.start(for:) again and that resets the assignments for the logged-in context. Were you able to get the treatment variation from your testing? Maybe we should consider this AB experiment in the logged-in context instead, since merchants can only install Jetpack after logging in?

[selanthiraiyan]

@jaclync

Were you able to get the treatment variation from your testing?

I remember getting the treatment variation. I think I tested it incorrectly while raising my PR.

Were you able to get the treatment variation from your testing? Maybe we should consider this AB experiment in the logged-in context instead, since merchants can only install Jetpack after logging in?

I started treating the experiment as logged-in, but so far I am not able to assign the "treatment" variation to my simulator's anonId (device ID).
I tried to debug and noticed that anonId is nil in func getAssignments(completion: @escaping (Assignments?) -> Void) while trying to reload experiment data after logging in. 🤔
I will keep looking into this. Meanwhile, kindly let me know if you can successfully test this by assigning the treatment variation.

[jaclync]

I tested again with the logged-in paragraph in the assignments API FG linked in the PR description, and it worked for my a8c WPCOM account (somehow it didn't work using a bearer token for a non-a8c account). I verified that the assignments from the API have the treatment variation set for the Jetpack experiment. However, the variant in the app use case stays control because the assignments from the API weren't saved to UserDefaults since the ExPlat instance was deallocated.

After looking into the deallocation of ExPlat a bit, it looks like ExPlat.shared is overridden whenever ExPlat.configure is called. When launching the app in the logged-in state, it's called first with an anon ID and then again in the authenticated state which reset ExPlat.shared so the previous instance was deallocated. The assignments API call was made with the first ExPlat instance, so the assignments value couldn't be saved for access later.

I haven't looked into a solution yet, maybe we'll have to make sure the ABTest assignments API call is made only with the authenticated ExPlat? I also wonder if this might have caused the biased A/A test that we saw earlier pbxNRc-1QS-p2#comment-3819.

[jaclync]

nit: abTest might be redundant in the case name?

[jaclync]

WDYT @selanthiraiyan?

[selanthiraiyan]

Sorry, I missed this. Done in 2121554

[itsmeichigo]

Thank you, @jaclync for looking into the issue and sharing the observation! I'm trying to understand the problem here:

I verified that the assignments from the API have the treatment variation set for the Jetpack experiment. However, the variant in the app use case stays control

Does it mean you can get the treatment variation after Sharma sets the Jetpack experiment to logged-in?

After looking into the deallocation of ExPlat a bit, it looks like ExPlat.shared is overridden whenever ExPlat.configure is called. When launching the app in the logged-in state, it's called first with an anon ID and then again in the authenticated state which reset ExPlat.shared so the previous instance was deallocated. The assignments API call was made with the first ExPlat instance, so the assignments value couldn't be saved for access later.

From my understanding, the problem is not about deallocation because the assignments are saved to UserDefaults after getting a response from getAssignments. The issue is that ExPlat was first configured with an anon ID, and then with a token - so the assignment for the same experiment might change. We want to use each experiment in only an unauthenticated or authenticated state, not both.

With that said, if we want to install Jetpack only in the authenticated state, Sharma's fix should fix the issue with the incorrect variant. I'm concerned if we want to support native Jetpack installation for site credential login in the next iteration - we'll need a new experiment for that then.

I also wonder if this might have caused the biased A/A test that we saw earlier

If my above understanding is correct, I think the fix for the crossovers is to reconfigure ExPlat again with anon ID when the user logs out of the app. This ensures the user gets the exact variation they saw before logging in. I encountered once that I got the different variant after logging out of the app, so doing that should fix the problem. WDYT?

[jaclync]

Does it mean you can get the treatment variation after Sharma sets the Jetpack experiment to logged-in?

From my understanding, the problem is not about deallocation because the assignments are saved to UserDefaults after getting a response from getAssignments.

I got the treatment variation when inspecting the assignments when the API result comes back here. And then when ExPlat is handling the assignments to save to UserDefaults, it early returned because self was deallocated. Therefore, I always got the control variation from ABTest.abTestNativeJetpackSetupFlow.variation because the assignments weren't saved to UserDefaults.

Just a note that I tested this when launching the app in the logged-in state to repro the issue more easily, instead of launching the app in the logged-out state and then logging in as in the PR testing steps.

If my above understanding is correct, I think the fix for the crossovers is to reconfigure ExPlat again with anon ID when the user logs out of the app. This ensures the user gets the exact variation they saw before logging in. I encountered once that I got the different variant after logging out of the app, so doing that should fix the problem. WDYT?

Other than the crossover issue, in pbxNRc-1QS-p2#comment-3819 (under Bias in performance) Aaron also noticed a bias toward the control variant in the A/A test results while no bias is expected. I wonder if this deallocation issue might be the reason since the control variant is used when no value is found in the UserDefaults assignments. I can look into a fix for the deallocation issue in a bit.

---

@itsmeichigo were you able to get the treatment variation following the testing steps with the latest commit? If so, please also feel free to go ahead with approval and merge the PR for the release.

[itsmeichigo]

were you able to get the treatment variation following the testing steps with the latest commit? If so, please also feel free to go ahead with approval and merge the PR for the release.

I could not test it because I don't have access to sandbox - I'll have to request that soon. So we'll need your approval anyway 😅

[jaclync]

Works as expected now! 🎉 I can get the treatment variant consistently.

Out of the scope of this PR, I wonder if it's worth creating another 2 A/A tests for logged-out and logged-in states for the same release to see if it fixes the bias towards the control variant mentioned in pbxNRc-1QS-p2#comment-3819?

[jaclync]

WDYT @selanthiraiyan?