Simple AB-Testing framework (support for Parsley)
ActionScript
Switch branches/tags
Nothing to show
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
example
libs
script
src-parsley/net/wooga/abtest/parsley
src-test
src/net/wooga/abtest
.gitignore
README.md

README.md

AS3 AB-Test

This library provides a way to handle AB tests within flash. Initially the framework selects the proper variants for a given user id. At runtime you can either ask the framework for the proper test value or you can use Parsley Metadata-Tag to get it injected automatically.

Use cases for this library:

  • Test certain configurations/functionalities/graphics only for certain users
  • Avoid AB-Test logic spread all over your application

Dependencies

This library is compiled against parsley-flash-2.4.1.swc and spicelib-flash-2.4.0.swc. The SWCs are included in the download file. The library should be compatible with older versions of Parsley/Spicelib. More information regarding Parsley and Spicelib can be found here.

Known issues

This framework calculates the sufficient variant once in the beginning. This is especially important for the Parsley extension as it injects the proper values. While this is a benefit in most cases it won't work for applications where the id used for determining the variant changes at runtime (e.g. the user can logout/login). This would only work if you setup a new Parsley context each time. As we use this library for Facebook games this is not an issue for us. The user always stays the same during one session

Usage

Download

Download the latest version from the downloads page. The zip file contains all necessary SWC files and the asdocs. Put the SWC files in your projects libs folder and/or add them to the classpath of your project.

AB-Testing introduction

AB-Testing is used to validate features or changes with only a small group of users. Then you can track how well the change performs for this small group of people and decide if you want to enable it for more/all users or discard the change.

Lets assume the following example: A new tooltip is introduced. 25% of all users should see the new tooltip, the rest (75% still the old one).

Because you always want to deliver the same variant for the same users you have to distinguish the variant on something user-specific. Usually the user id is unique and never changes. For Facebook games the users facebook-id can be used to determine the variant.

Based on how you want to distribute your variants on your user base you can now identify a modulo value. For our example a modulo of 4 would be perfect because would divide our users into 4 groups (each 25%). Here are some examples:

user-id		modulo		group

100			4			0
123			4			3
45			4			1
262			4			2
4563		4			3

Now all our users are mapped to one of the four groups. But in our example we only want 25% to see the new tooltip. So we need some configuration that defines which test group should apply to which variant. Something like this:

groups		variant		
            
2			A			
0,1,3		B			

With this configuration we can now distinguish the variant for each user.

Configuration

The configuration usually happens via xml or json file. Either the Backend or Client loads this file and then decides upon the users id which variant to select. This framework provides one default xml configuration solution for the client. But it is very easy to write your own configuration mechanism. The only thing you have to do is implementing the ABConfigurator interface:

import net.wooga.abtest.config.ABConfigurator;
import net.wooga.abtest.model.ABData;

public class CustomConfigurator implements ABConfigurator
{
	private var abData:ABData = new ABData();

	// Pass in any data necessary for configuration
	public function CustomConfigurator()
	{
		// Select the appropriate variant for the user
		
		// Set proper data within the abData object
		abData.addTestData("testId", "testValue");
	}

	public function getData():ABData
	{
		// The abData will be requested by the framework on initialization
		return abData;
	}
}

To use you custom configuration initialize the ABTest class with it:

var abConfigurator:ABConfigurator 
	= new CustomConfigurator();

var abTest:ABTest = new ABTest();
abTest.initialize(abConfigurator);

Using the default configuration mechanism

If you want to use the xml configuration that is provided by the framework, your Configuration file needs to have a certain structure (xsd is following!). The following snippet is taken from the example that you can find in the example folder.

<?xml version="1.0" encoding="UTF-8"?>
<tests>
	<test id="test1" default="20" modulo="4">
		<variant value="5" testGroups="1"/>
		<variant value="10" testGroups="2,3"/>
	</test>
	<test id="useNewFeature" modulo="2" default="false">
		<variant value="true" testGroups="1"/>
	</test>
</tests>

The configuration consists of a number of test nodes (<test></test>). Each test accepts the following attributes:

  • id: unique id for the test
  • modulo: to divide into user-groups
  • default: fallback value if none of the variants matches

As child node you can define any amount of variants (<variant/>). Each variant accepts these attributes:

  • value: value for this variant (any string)
  • testGroups: comma-separated list of groups

If there is any group not matched in the variant it will select the default value from the <test> node.

For using the default configuration mechanism use the XMLConfigurator class. Note, that it expects an already loaded xml object as constructor argument. So you have to make sure that the xml file is loaded upfront.

var abConfigurator:XMLConfigurator 
	= new XMLConfigurator(currentUserId, new XML(urlLoader.data));

abTest = new ABTest();
abTest.initialize(abConfigurator);

What happens now is, that the XMLConfigurator class will iterate over all tests and select the appropriate variant for the assigned currentUserId. To see this snippet in action check out the example application in the example folder.

Configuration from Backend

Sometimes it makes sense to do this calculation on the backend side because it also needs to know about certain AB-Tests. In this scenario the client implementation of the ABConfigurator would be fairly simple, as it only has to assign the pre-selected data from the backend to the ABData object.

Basic usage (instance)

After the ABTest is initialized you can directly retrieve test data by id:

var abTest:ABTest = new ABTest();
abTest.initialize(abConfigurator);

var testValue:String = abTest.getTestValue("testId");

Basic usage (static)

In order to make access easier there is also a static access available:

ABTest.getTestValue("testId", "fallbackValue");

As you can see, when using the static access you have to define also a fallback value. This is necessary in order to make sure your application also works when the ABTest with this id got deleted from your configuration.

Now your application won't break but it would be still nice to see whether a certain ABTest is unused. For this purpose you can define a third boolean property warn:

ABTest.getTestValue("testId", "fallbackValue", true);

When set to true (default) the framework will generate a log message of type WARNING using Spicelibs logging framework.

Injection (Parsley)

If you are using Parsley 2 for your project, you can utilise the optional Parsley extension. It allows to get the proper test value injected by the framework. This reduces the amount of ABTesting code to a minimum. Your classes get much more readable as they don't get messed up by ABTest logic. To make the extension work you have to initialise it properly when setting up the context:

// create the context builder
var contextBuilder:ContextBuilder 
	= ContextBuilder.newSetup().newBuilder();

// initialize the Parsley ABTest extension by passing in 
// the formerly created abTest instance
ABTestSupport.initialize(contextBuilder, abTest);

// go on with context initialization
contextBuilder.config(
		ActionScriptConfig.forClass(ExampleContext)
	).build();

What happens now it, the framework will search for all [ABTest] metadata tags in your managed classes. Whenever it finds one it will inject the previously set test value into the annotated property.

This is how a complete metadata tag declaration may look like:

[ABTest(id="testId",fallback="fallbackValue",warn="false")]
public var test:String;

The id has to match one of the id's stored in the ABData object. If it can not find data for this id, it will use the value defined as fallback. If you set the warn flag to false you won't see an log entry in case it couldn't find a test for the id.

However most of the time you won't have to declare all this properties. This is how it could also look like:

[ABTest(fallback="fallbackValue")]
public var testId:String;

Note that this time the property is called testId instead of test. Because there is no id attribute specified in the metadata tag it will reflect on the property name and take this one as test-id. As the warn attribute is anyway optional (default = true) it can be omitted. The fallback should always be defined so the application still works properly if though the test got removed from the configuration. A warning will show up in the log console anyway then.

Sometimes strings are not sufficient for AB-Testing. You may be want to have different classes that get executed or just set a boolean flag. The good new: Different types are automatically converted. In the following example we use class injection:

[ABTest(fallback="com.wooga.example.ExampleClass")]
public var testClass:IExampleClass;

Here we expect an instance of type IExampleClass (interface). This interface is implemented by different classes, one of them is com.wooga.example.ExampleClass (the fallback). Based on the variant the framework automatically creates the proper instance and assigns it. It will also add it as a dynamic object! So you don't have to do this separately.

Note: In order to get the concrete instance assigned, you have to make sure it is included in your sources. Because the creation happens dynamically at runtime, it won't include the classes by default. Instead it would cause an error on initialisation. In order to enforce inclusion you have put something like the following somewhere in your code:

var forceInclusion:Array = [ExampleClass, ExampleClassB];

Beside Classes, primitives like Numbers and Booleans are also supported:

[ABTest(fallback="false")]
public var useNewFeature:Boolean;

Even though you declared in the configuration a String, Parsley will convert it into a proper Boolean value automatically.

Robotlegs support

So far there is no Robotlegs support. It would be nice if anyone could contribute this. But in the meantime you can always make use of the static access via ABTest.getTestValue() method.

Change log

0.0.1 (2011/12/21)

  • [Added] AB-Test logic
  • [Added] Static access for results
  • [Added] Parsley support via [ABTest] metadata tag