Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

A/B testing, Split tests #459

Open
anonymous-piwik-user opened this Issue · 23 comments

8 participants

@anonymous-piwik-user

Which of our landing pages are the most successful and convert our visitors better?

How often did we wonder if we have the right words and images on our landing page (the page where visitors arrive from your advertising)?[What if changing just one image on the page could increase the visitor conversion rate several times?[[BR]([BR]])] We will never know this without testing.

The good news is that there is a technology that can help us quickly determine the best version of our landing page. It is called Split-Testing and it should be implemented in Piwik.

Piwik should randomly show different versions of the same landing page to our visitors. It should make sure that each version of the page is displayed the same number of times overall and it should even display the same version of the page to the same visitor (to make their experience with our site consistent).

With split-tests we should be able to quickly see which version of the page has a good conversion rate and which one is just a waste of the advertising money. Get rid of the unsuccessful version, keep the good one and our overall conversion will increase.

@robocoder
Collaborator

WebSite Optimizer is being folded into GA.

@mattab
Owner

I think it was reasonnable for us to recommend using Website Optimizer, but now that they are forcing users to use GA for split testing, we should consider implementing similar functionality in Piwik :)

Random thoughts

@mattab
Owner

It would be fantastic to have A/B testing feature built in Piwik, as a plugin. If you are interested and could help us make it happen, please email at hello@piwik.org!

@nkuehn

my two cents in case you consider implementing A/B testing support (which would be awesome):

An extremely powerful way to realize the A/B testing capabilities but also much more would be to add a generic ability to compare segments. The new segment editor is awesome and could be leveraged even more.

A/B variant tags would then just be custom variables (typically visit-scope) like any other.

A feature like Google Analytics' capability to actively control which users get which variant from inside the analytics system is just for convenience, but not a core requirement IMHO. One can do the actual traffic split on the website server (or loadbalancer) or in the webisteJavascript as well.

@mattab
Owner

Facebook has released their A/B testing tool: http://facebook.github.io/planout/ https://github.com/facebook/planout - it's written in python. Maybe we could get some inspiration from them.

@pattonwebz

I'm in agreement with nkuehn about it being unnecessary to have the ability to control variants from inside the analytics system.

I also don't think that the analytics platform is the best place to be running these tests. I'd think that the purpose of the analytics would be to track the test results - and not actually be the medium for defining or conducting the test at all.

I've been tracking split-test campaigns in GA for a while now using Events. Earlier today Piwik also introduced Event tracking (http://piwik.org/docs/event-tracking/) which I was then able to integrate with my exiting split testing framework (based on this javascript library on github: https://github.com/jamesyu/cohorts/).

Using almost exactly the same code as I use for GA tracking I am now able to track split-test campaigns with Piwik through with the Javascript API and Events. I'm using the following code which reports the same data to both GA and Piwik. First line is passing data to GA, second to Piwik.

_gaq.push(['_trackEvent', category, action, opt_label, opt_value, int_hit]);
_paq.push(['trackEvent', category , action, opt_label, opt_value]);

That code is called onInitialization of the test (consistency is handled by cookies before test initialization) to track how many times each test runs and then it's allso called a second time onClick passing a different label so that you can segment views from clicks.

@mattab
Owner

Kissmetrics A/B testing product is interesting also: http://blog.kissmetrics.com/kissmetrics-ab-test-report/

@mattab
Owner

I also don't think that the analytics platform is the best place to be running these tests. I'd think that the purpose of the analytics would be to track the test results - and not actually be the medium for defining or conducting the test at all.

I agree that it is not the "best place", as Piwik is best at reporting analytics. So it will be possible to use Piwik to only measure A/B testing results, and not define the variants. Users could use their favorite A/B testing tool to define and manage tests and variants, and send the data to Piwik.

But we definitely have to also include a basic version of this functionnality within Piwik. If we don't, users will be tied to an external provider, which is not ideal. Our vision is to empower users with the toolset to measure and make changes on their website. It's clear to me that running A/B tests (create variants, serve them, measure performance, report results) is a key part of this vision.

@pattonwebz

Hey Matt,

At least we agree that the 'best place' isn't directly within the Piwik platform, but it obviously wouldn't be a breaking change for me if it did include some functions for implementing tests.

I have seen the roadmap showing that A/B testing is to be implemented into the core - which is how I found this ticket - and I don't necessarily think it's a bad idea.

My primary reason for stating objections is not to prevent it being added (because it would be a greatly welcomed featured for a lot of users) but rather to note that some users would rather make use of a platform independent testing model that does not tie you directly to any one anayltics platform or the other. As such it would mean that the testing would need to be turned on/off at the provider end to prevent bloat in the tracking code for users who don't want it.

I get that having to use an external provider is not ideal but self-hosted libraries are also an option to negate the requirement of an external provider.

I guess what I'm really saying is that platform independence is important FOR ME but I get that it's not as important for others.

As I stated I already run tests and track them with Piwik using Events and I'd actually love to be able to help shape the integration of A/B testing to the core - which is why I commented on this ticket in the hope that someone from the core team would see it and I'd be able to make suggestions and push for my choice of perfect match features in they system ;)

@mattab
Owner

Good we're having this discussion, it's useful!

Ideas about how we could architecture an A/B testing solution in Piwik: I think we could break down in several modules:

  1. a tool to define tests and their variants, and serve them in the web page or app.
  2. the ability to measure with Piwik the interactions and changes in conversion rates, for each test's variant. It would work for tests created with other A/B tools that offer Javascript APIs to get the tests and variants (such as optimizely).
  3. Ideally, the A/B testing feature would have an algorithm that decides when to stop running the test, and picks a variant that is significantly performing better (if any can be picked). It could also automatically display the winning variant to display on the site by default (optional). Ref: post about an epsilon greedy algorithm to generate variant probability in a bit clever.

Keep feedback coming!

@mattab
Owner

Interesting also: http://bentilly.blogspot.co.nz/2012/10/ab-testing-scale-cheat-sheet.html

Most Winning A/B Test Results are Illusory On the importance of running the power calculation: What is sample size needed? Indicate how long will the test likely take before reaching sample size.

@nkuehn

Hi erveryone, great to see that there's a lot of interest. What do you think about splitting the ticket into two:

  1. one about the proposed feature that allows controlling the test run from inside piwik (I agree with Matt that there's relevant market pressure to have some kind of feature like this, no matter how big or small it is - I personally don't need it, but that's not relevant)
  2. one about the necessary analysis features (especially visualizing differences between segments / test groups instead of having to switch back and forth manually), which are required anyways, no matter which way you run the tests (this new piwik feature, homebrew, optimizely, ... you name it).

The "only" thing that both definitely need is a conceptual decision on how the A/B comparison is realized in the piwik core (e.g. visual segment comparison vs. a specialized feature with own database structures vs. ... ). Probably the hard part...

@williampatton: I personally think that custom session-scope variables are more suitable to tag the visitors into A/A2/B/etc than events (conversion is the typical success aspect and conversion is usually session-scope, which events aren't). Unfortunately their number is limited as far as I understand the Piwik API.

@mattab
Owner

Interesting article: What you should know about the Optimizely vulnerability which discusses the privacy implications of A/B testing implementations, and how, we should aim to avoid 'leaking' A/B tests details in the javascript of A/B testing users.

@tassoman

I agree that having a full AB Testing feature would be having too much inside Piwik.

In my opinion would be enough having an super easy way of integrating Piwik's Events feature, or Piwik's Custom Variables into a third parties A/B Testing tool, just like a pluggable component linking directly Piwik's APIs filling in the data will be measured.

@mattab mattab added Major and removed c: Core Critical labels
@mattab
Owner

@nkuehn

one about the necessary analysis features (especially visualizing differences between segments / test groups instead of having to switch back and forth manually)

we developed this feature and have released it open source. for now it's available here: https://piwik.pro/plugins/comparison-dashboard

but in the near future it will be in the marketplace. It adds a new dashboard where you can compare at the same time three segments in 3 columns view (and add any widget there). This is cool stuff!

@mattab
Owner

In my opinion would be enough having an super easy way of integrating Piwik's Events feature, or Piwik's Custom Variables into a third parties A/B Testing tool, just like a pluggable component linking directly Piwik's APIs filling in the data will be measured.

current experiment picked by A/B testing logic will be stored in a cookie and in a custom variable in piwik visit and page rows, and we may also use it to segment the report data for each experiment. we plan to launch a crowdfunding campaign within one month!

@mattab mattab added the c: Platform label
@Tobias-Conrad

Hi what is the status? How could A/B Testing be used?

@nkuehn

FYI: I have a new Job and therefore can' participate with ideas and testing based on real world data any more. The comparison dashboard looks very nice as far as the screeshot can tell. Hopefully you get the funding campaign completed!

@mattab mattab modified the milestone: Short term, Mid term
@marclaporte

20 lines of code that will beat A/B testing every time
http://stevehanov.ca/blog/index.php?id=132

What’s Happening In The A/B Testing Market?
https://blog.builtwith.com/2013/07/19/whats-happening-in-the-ab-testing-market/

Seeking Edge, Websites Turn to Experiments ; Optimization technology is reshaping publishers’ decision-making process—and the Web itself.
http://www.technologyreview.com/news/523671/seeking-edge-websites-turn-to-experiments/

"On 1 June 2012, Google announced that GWO as a separate product would be retired as of 1 August, and some of its functionality would be integrated into Google Analytics as Google Analytics Content Experiments. Google Website Optimizer allowed webmasters to test an alternative version of an entire page, known as A/B testing — or test multiple combinations of page elements such as headings, images, or body copy; known as Multivariate testing. This tool was part of Google Analytics, though accessed through a different user interface. As it was part of Google Analytics, it used the Google analytics scripts."
http://en.wikipedia.org/wiki/Google_Website_Optimizer

@Tobias-Conrad

ticket opened 2008, added to major 2014,
may it is time to give a easy solution made by piwik.
like https://clicky.com/help/faq/features/split-tests/how

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.