Skip to content

Commit

Permalink
docs: unexpected results in abn test troubleshooting guide (#6153)
Browse files Browse the repository at this point in the history
https://linear.app/unleash/issue/SR-374/troubleshooting-guide-my-feature-flag-is-returning-unexpected-results

Adds a quick troubleshooting guide that covers some common pitfalls when
running A/B tests.
  • Loading branch information
nunogois committed Feb 8, 2024
1 parent fa33527 commit 5d4c750
Show file tree
Hide file tree
Showing 4 changed files with 36 additions and 1 deletion.
@@ -0,0 +1,34 @@
---
title: My feature flag is returning unexpected results in my A/B/n test
---

Occasionally, users might come across a situation where a feature flag is returning unexpected results when running an A/B/n test.

The first thing to look into is your feature configuration.

If you're unsure whether the feature flag is being properly returned, you can go through the steps in this troubleshooting guide before proceeding: [My feature flag is not returned in the Frontend API/Edge/Proxy](/using-unleash/troubleshooting/flag-not-returned.md).

1. If you're using a [gradual rollout activation strategy](/reference/activation-strategies#gradual-rollout) be sure to set the rollout percentage to the percentage of your user base you would like to target for the test. If you would like to target 100% of your user base on your A/B/n test, confirm that your rollout percentage is set to 100%. Alternatively, you may want to consider using a [standard strategy](/reference/activation-strategies#standard) instead, since that means "active for everyone". This adjustment guarantees that all users participate in the A/B/n test, irrespective of the division ratio, be it an even 50-50 split, for a typical A/B test with 50% for Group A and 50% for Group B, or any other distribution that you configure in your variants.
2. When using a [gradual rollout strategy](/reference/activation-strategies#gradual-rollout), be mindful of the [stickiness](/reference/stickiness) value. If you're using default stickiness, confirm that either `userId` or `sessionId` are part of your context to ensure consistent results. Besides, if the provided context does not include the field used in the stickiness configuration, the gradual rollout strategy will be evaluated to `false` and therefore it will not be returned by the API.
3. Ensure that your variants are correctly configured. You can refer to [feature toggle variants](/reference/feature-toggle-variants). For example, if you would like to run a simple 50-50 A/B test, then your variants should look similar to this:

![An example of variants configured for an A/B test](/img/troubleshooting-flag-abn-test-unexpected-result-variants.png)

4. Double check that your code is correctly handling the feature flag variant response. You can refer to your SDK documentation for more information on how to handle feature flag variants. For example, if you're using the [Unleash React SDK](/reference/sdks/react), you can follow the [check variants](/reference/sdks/react#check-variants) section of the documentation. Given the example variants above, this could look like the following:

```tsx
import { useVariant } from '@unleash/proxy-client-react';

export const TestComponent = () => {
const variant = useVariant('ab-test-flag');

if (variant.name === 'A') {
return <AComponent />;
} else if (variant.name === 'B') {
return <BComponent />;
}
return <DefaultComponent />;
};
```

Feature activation strategies can be combined in different ways, which may lead to complex scenarios. If you're still not seeing the results you expect, try using the [Playground](/reference/playground.mdx) to verify that the feature is properly configured and responding as expected.
Expand Up @@ -15,4 +15,4 @@ The first thing to look into is to validate that the feature is well configured
1. **Access to a discrete list of projects** - Tokens with a leading set of square brackets (empty) will be given access to a subset of projects in a particular environment. The token will look similar to the following: `[]:production:xyz123etc...`. Which projects the token has access to can be found on the API Tokens page in the Unleash admin UI.
1. **Single project access** - Tokens that lead with a project name are bound to the specified project and environment. For example, `my_fullstack_app:production:xyz123etc...` will only have access to flags in the "my_fullstack_app" project as set in the production environment.
1. When using a **gradual rollout** strategy, be mindful of the **[stickiness](/reference/stickiness)** value. When evaluating a flag, if the provided context does not include the field used in the stickiness configuration, the gradual rollout strategy will be evaluated to `false` and therefore it will not be returned by the API.
1. Feature activation strategies may have constraints, segments, and rules that can be combined in different ways that can lead to complex scenarios. Try using the [Playground](/reference/playground.mdx) to verify that the feature is properly configured and responding as expected.
1. Feature activation strategies can be combined in different ways, which may lead to complex scenarios. Try using the [Playground](/reference/playground.mdx) to verify that the feature is properly configured and responding as expected.
1 change: 1 addition & 0 deletions website/sidebars.js
Expand Up @@ -468,6 +468,7 @@ module.exports = {
'using-unleash/troubleshooting/feature-not-available',
'using-unleash/troubleshooting/flag-exposure',
'using-unleash/troubleshooting/flag-not-returned',
'using-unleash/troubleshooting/flag-abn-test-unexpected-result'
],
},
'generated/unleash-edge',
Expand Down
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 5d4c750

Please sign in to comment.