A/B tests on donate page

[ajparsons]

Following on from donation analysis - this is the key ticket.

Basically we want to be able to run A/B tests on the location and content of the donate link on the WDTK requests page. @zarino says this is actually super easy and customisable through google analytics - so this is a ticket to a) ask someone to do this b) draw out any additional problems/info needed.

[zarino]

Here’s an example of the JavaScript involved with a recent A/B test we did on the People’s Assembly homepage: https://github.com/mysociety/pombola/pull/2269/files

It pretty much boils down to:

window.cxVariation = cxApi.chooseVariation();
if(window.cxVariation === 1){
    // Change content on page
}

Then you set up your success criteria in the Content Experiment section of Google Analytics, and you’re away.

[ajparsons]

For future reference, I have created issue stubs for the first three experiments in the alaveteli-experiments repo

• Does placement of donation button change donation click-through? alaveteli-experiments#115 – varying placement of banner
• Does varying the text of the donation call to action increase click through alaveteli-experiments#116 – varying text of banner
• Does varying the text and number of buttons on the donation call-to-action change click-through? alaveteli-experiments#117 – varying buttons on banner

[wrightmartin]

Can we add one to this for changing the design? I think the red box design is a big miss here - I'd guess it gets mistaken for an error message? I assume in the spirit of this ticket we would A/B test a design change?

[ajparsons]

Can we add one to this for changing the design?

Created!

• Does changing the design of the donation call to action increase clickthrough? alaveteli-experiments#118 – varying design of banner

Once the capacity is in place we can informally work out what we believe would make the biggest impacts and work out a test order. Some like design, I'd be tempted to change from the start and then test later on if the original was correct.

[wrightmartin]

Quick redesign of what the donate call could look like

[zarino]

@wrightmartin Is there a danger of it starting to look too much like a sidebar advert? The sort of thing we’ve all been trained to ignore ;-)

[wrightmartin]

@wrightmartin Is there a danger of it starting to look too much like a sidebar advert? The sort of thing we’ve all been trained to ignore ;-)

Yep - but it's okay, because we're testing that too mysociety/alaveteli-experiments#115

[garethrees]

So far:

• Does placement of donation button change donation click-through? alaveteli-experiments#115 – varying placement of banner

This made an improvement so we've made it permanent

• Does varying the text of the donation call to action increase click through alaveteli-experiments#116 – varying text of banner

Donate £n Now vs Donate Now doesn't seem to make much difference

So the next thing to test is:

• Does varying the text and number of buttons on the donation call-to-action change click-through? alaveteli-experiments#117 – varying buttons on banner

[garethrees]

psyche.co have quite a nice prominent banner. I like the user testimonial. I wonder if we could collect a bunch of these from Twitter?

Their parent site aeon.co has similar. I love that they're clear about their mission, and I can see why that creates a resonance with potential donors.

We should integrate better wording (mysociety/alaveteli-experiments#116 (comment)) into our banners.

[HelenWDTK]

@ajparsons (or anyone else), has this been completed now to the extent this issue can be closed?

[ajparsons]

I mean we're not doing it, probably not going to do it soon, if we do we can reopen this.