Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

A/B testing, Split tests #459

Open
anonymous-piwik-user opened this Issue · 25 comments

9 participants

@anonymous-piwik-user

Which of our landing pages are the most successful and convert our visitors better?

How often did we wonder if we have the right words and images on our landing page (the page where visitors arrive from your advertising)?[What if changing just one image on the page could increase the visitor conversion rate several times?[[BR]([BR]])] We will never know this without testing.

The good news is that there is a technology that can help us quickly determine the best version of our landing page. It is called Split-Testing and it should be implemented in Piwik.

Piwik should randomly show different versions of the same landing page to our visitors. It should make sure that each version of the page is displayed the same number of times overall and it should even display the same version of the page to the same visitor (to make their experience with our site consistent).

With split-tests we should be able to quickly see which version of the page has a good conversion rate and which one is just a waste of the advertising money. Get rid of the unsuccessful version, keep the good one and our overall conversion will increase.

@robocoder

WebSite Optimizer is being folded into GA.

@mattab
Owner

I think it was reasonnable for us to recommend using Website Optimizer, but now that they are forcing users to use GA for split testing, we should consider implementing similar functionality in Piwik :)

@mattab
Owner

It would be fantastic to have A/B testing feature built in Piwik, as a plugin. If you are interested and could help us make it happen, please email at hello@piwik.org!

@nkuehn

my two cents in case you consider implementing A/B testing support (which would be awesome):

An extremely powerful way to realize the A/B testing capabilities but also much more would be to add a generic ability to compare segments. The new segment editor is awesome and could be leveraged even more.

A/B variant tags would then just be custom variables (typically visit-scope) like any other.

A feature like Google Analytics' capability to actively control which users get which variant from inside the analytics system is just for convenience, but not a core requirement IMHO. One can do the actual traffic split on the website server (or loadbalancer) or in the webisteJavascript as well.

@mattab
Owner

Facebook has released their A/B testing tool: http://facebook.github.io/planout/ https://github.com/facebook/planout - it's written in python. Maybe we could get some inspiration from them.

@pattonwebz

I'm in agreement with nkuehn about it being unnecessary to have the ability to control variants from inside the analytics system.

I also don't think that the analytics platform is the best place to be running these tests. I'd think that the purpose of the analytics would be to track the test results - and not actually be the medium for defining or conducting the test at all.

I've been tracking split-test campaigns in GA for a while now using Events. Earlier today Piwik also introduced Event tracking (http://piwik.org/docs/event-tracking/) which I was then able to integrate with my exiting split testing framework (based on this javascript library on github: https://github.com/jamesyu/cohorts/).

Using almost exactly the same code as I use for GA tracking I am now able to track split-test campaigns with Piwik through with the Javascript API and Events. I'm using the following code which reports the same data to both GA and Piwik. First line is passing data to GA, second to Piwik.

_gaq.push(['_trackEvent', category, action, opt_label, opt_value, int_hit]);
_paq.push(['trackEvent', category , action, opt_label, opt_value]);

That code is called onInitialization of the test (consistency is handled by cookies before test initialization) to track how many times each test runs and then it's allso called a second time onClick passing a different label so that you can segment views from clicks.

@mattab
Owner

Kissmetrics A/B testing product is interesting also: http://blog.kissmetrics.com/kissmetrics-ab-test-report/

@mattab
Owner

I also don't think that the analytics platform is the best place to be running these tests. I'd think that the purpose of the analytics would be to track the test results - and not actually be the medium for defining or conducting the test at all.

It is not the "best place" if you are already using a A/B testing tool, but many people are only using Piwik on their website as "third party javascript" and would enjoy basic A/B tests feature in Piwik.

Maybe we could make it possible to use Piwik to measure A/B testing results from another tool? Users could use their favorite A/B testing tool to define and manage tests and variants, and send the data directly to Piwik (or Piwik fetch it from their API).

Our vision is to empower users with the toolset to measure and make changes on their website. Running A/B tests (create variants, serve them, measure performance, report results) is a key part of this vision :+1:

@pattonwebz

Hey Matt,

At least we agree that the 'best place' isn't directly within the Piwik platform, but it obviously wouldn't be a breaking change for me if it did include some functions for implementing tests.

I have seen the roadmap showing that A/B testing is to be implemented into the core - which is how I found this ticket - and I don't necessarily think it's a bad idea.

My primary reason for stating objections is not to prevent it being added (because it would be a greatly welcomed featured for a lot of users) but rather to note that some users would rather make use of a platform independent testing model that does not tie you directly to any one anayltics platform or the other. As such it would mean that the testing would need to be turned on/off at the provider end to prevent bloat in the tracking code for users who don't want it.

I get that having to use an external provider is not ideal but self-hosted libraries are also an option to negate the requirement of an external provider.

I guess what I'm really saying is that platform independence is important FOR ME but I get that it's not as important for others.

As I stated I already run tests and track them with Piwik using Events and I'd actually love to be able to help shape the integration of A/B testing to the core - which is why I commented on this ticket in the hope that someone from the core team would see it and I'd be able to make suggestions and push for my choice of perfect match features in they system ;)

@mattab
Owner

Good we're having this discussion, it's useful!
Here are some thoughts on how it a new A/B Plugin could be broken down:

  1. let user define A/B tests and their variants via new UI and API. Variations include:
    • URL variations of same page new (mvp?),
    • let user create tests variants by inputting a HTML text (mvp?),
    • advanced UI to create test variants directly within the website like commercial A/B testing tools do very nicely (not in mvp)
  2. Javascript tracker (client side)
    • the Javascript tracker fetches the tests variants for this the web page or app,
      • (we need to let user in JS know about which A/B tests are running on the site. let's think of how we can avoid leaking all details of running A/B tests in the process?),
    • decides how to assign a visitor to a given test and given variant based on the variant probability,
      • maybe we start with a simple algorithm eg. 0.5 probability of A and B variants
      • (we may need to process the probabilities server side in case some of the variants will need to be given more or less traffic)
    • selected test variant info is persisted in a new first party cookie,
      • a user will be always shown the same variant (using this cookie we know which variant was served before),
    • Maybe we could reuse some code or ideas from this project for JS API
  3. Tracking API (server side)

    • Store the tests and variants viewed by a visitor, for each visit and goal conversion
      • do we also need to store variants for each action / page/ event?
  4. Reporting:

    • process the KPIs for each A/B test such as: % change in Actions per visit, Time on site...
    • process Conversion rate of each A/B test variant for each Goal known in Piwik (eg. Goal "Download brochure" conversion rate increased 50% in test variant A")
    • create an overview report listing most important metrics and in information, and also let user view all % changes in KPIs for each running A/B tests,
    • Let user know when test is finished, eg. this website calculator http://www.experimentcalculator.com/#lift=20&conversion=5&visits=1000

Other notes & ideas about A/B plugin:

  • when disableCookies is used, maybe do not disable the a/b testing cookie
  • let user know when a A/B test has enough data to be statistically significant.
  • often (for low-med traffic websites) it may take several weeks to get statistically significant results, maybe we could notify users that a A/B test(s) is finished?
  • maybe we could automatically pick the winning variant when a test is marked as "statistically significant better" (useful if user cannot directly implement the changes in site or app) )
  • A/B tests could be available as Segment so users can see all Piwik reports segmented by experiments
    • this also would let users open in one click, a segment of visitors that were running in variant [abTestVariantName] in [abTestName]
  • maybe it could work for tests created with other A/B tools that offer Javascript APIs to get the tests and variants (such as optimizely).
  • maybe in Visitor log we could show which tests & tests variants were seen by a given user and/or given action.
  • maybe users already use other A/B tests tools where they create variant and just want to "import them in Piwik" to measure them... supporting this may be complicated)
  • random testing idea: we should run A/A experiments and check that the A/B testing engine does not declare one experiment as being "statistically significant" better!
  • (not mvp) a A/B test could be customised to be served only to a particular "high value" segment (based on a Custom Segment in Piwik) (optimizely does something like this)

Keep feedback coming!

(edited)

@mattab
Owner

Interesting also: http://bentilly.blogspot.co.nz/2012/10/ab-testing-scale-cheat-sheet.html

Most Winning A/B Test Results are Illusory On the importance of running the power calculation: What is sample size needed? Indicate how long will the test likely take before reaching sample size.

@nkuehn

Hi erveryone, great to see that there's a lot of interest. What do you think about splitting the ticket into two:

  1. one about the proposed feature that allows controlling the test run from inside piwik (I agree with Matt that there's relevant market pressure to have some kind of feature like this, no matter how big or small it is - I personally don't need it, but that's not relevant)
  2. one about the necessary analysis features (especially visualizing differences between segments / test groups instead of having to switch back and forth manually), which are required anyways, no matter which way you run the tests (this new piwik feature, homebrew, optimizely, ... you name it).

The "only" thing that both definitely need is a conceptual decision on how the A/B comparison is realized in the piwik core (e.g. visual segment comparison vs. a specialized feature with own database structures vs. ... ). Probably the hard part...

@williampatton: I personally think that custom session-scope variables are more suitable to tag the visitors into A/A2/B/etc than events (conversion is the typical success aspect and conversion is usually session-scope, which events aren't). Unfortunately their number is limited as far as I understand the Piwik API.

@mattab
Owner

Interesting article: What you should know about the Optimizely vulnerability which discusses the privacy implications of A/B testing implementations, and how, we should aim to avoid 'leaking' A/B tests details in the javascript of A/B testing users.

@tassoman

I agree that having a full AB Testing feature would be having too much inside Piwik.

In my opinion would be enough having an super easy way of integrating Piwik's Events feature, or Piwik's Custom Variables into a third parties A/B Testing tool, just like a pluggable component linking directly Piwik's APIs filling in the data will be measured.

@mattab mattab added Major and removed c: Core Critical labels
@mattab
Owner

@nkuehn

one about the necessary analysis features (especially visualizing differences between segments / test groups instead of having to switch back and forth manually)

we developed this feature and have released it open source. for now it's available here: https://piwik.pro/plugins/comparison-dashboard

but in the near future it will be in the marketplace. It adds a new dashboard where you can compare at the same time three segments in 3 columns view (and add any widget there). This is cool stuff!

@mattab
Owner

In my opinion would be enough having an super easy way of integrating Piwik's Events feature, or Piwik's Custom Variables into a third parties A/B Testing tool, just like a pluggable component linking directly Piwik's APIs filling in the data will be measured.

current experiment picked by A/B testing logic will be stored in a cookie and in a custom variable in piwik visit and page rows, and we may also use it to segment the report data for each experiment.

edit: interesting template: http://www.bloggerchica.com/test-design-the-official-doc/

@mattab mattab added the c: Platform label
@Tobias-Conrad

Hi what is the status? How could A/B Testing be used?

@nkuehn

FYI: I have a new Job and therefore can' participate with ideas and testing based on real world data any more. The comparison dashboard looks very nice as far as the screeshot can tell. Hopefully you get the funding campaign completed!

@mattab mattab modified the milestone: Short term, Mid term
@marclaporte

20 lines of code that will beat A/B testing every time
http://stevehanov.ca/blog/index.php?id=132

What’s Happening In The A/B Testing Market?
https://blog.builtwith.com/2013/07/19/whats-happening-in-the-ab-testing-market/

Seeking Edge, Websites Turn to Experiments ; Optimization technology is reshaping publishers’ decision-making process—and the Web itself.
http://www.technologyreview.com/news/523671/seeking-edge-websites-turn-to-experiments/

"On 1 June 2012, Google announced that GWO as a separate product would be retired as of 1 August, and some of its functionality would be integrated into Google Analytics as Google Analytics Content Experiments. Google Website Optimizer allowed webmasters to test an alternative version of an entire page, known as A/B testing — or test multiple combinations of page elements such as headings, images, or body copy; known as Multivariate testing. This tool was part of Google Analytics, though accessed through a different user interface. As it was part of Google Analytics, it used the Google analytics scripts."
http://en.wikipedia.org/wiki/Google_Website_Optimizer

@Tobias-Conrad

ticket opened 2008, added to major 2014,
may it is time to give a easy solution made by piwik.
like https://clicky.com/help/faq/features/split-tests/how

@Tobias-Conrad

@mattab just do it. why not finished?

@abishekrsrikaanth

This is a PHP based implementation of Facebook PlanOut. https://github.com/vimeo/ABLincoln

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.