Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
Already on GitHub? Sign in to your account
Begin Split Testing Bullets on Index Page #976
Conversation
harding
referenced this pull request
Jul 26, 2015
Closed
Build: Allow Rebuilding A Particular Commit #977
|
In discussion with @saivann about how we obfuscate IP addresses, I realized I need to make additional changes, so I'm temporarily closing this PR until it's ready for review again. |
harding
closed this
Jul 26, 2015
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
harding commentedJul 26, 2015
In aborted pull #972, I proposed to temporarily remove the three key points (bullet points) from the main page and to later replace them with different text that I thought had a lower probability of becoming untrue as Bitcoin grows and changes.
It was counter-argued that the points were important to making Bitcoin appealing so they should remain on the page, possibly after rewriting them better reflect the current situation or by adding asterisks (or links) pointing to qualifications.
This PR attempts to provide us with empirical evidence about how important the existing bullet points are by split testing them against similar points that I think are more likely to remain true for the foreseeable future.
Test Method
The test is only conducted on the English-language index page,
bitcoin.org/en/. The test does not use any Javascript or third-party analysis services.Instead, a cronjob[1] will be added to the build server to force rebuild the site at midnight every night, with all site builds on even days-of-the-year (1..365) rendering the current site bullet points (the control text) and odd days-of-the-year using revised bullet points (the test text).
Page loads from the webserver logs will be collected, as we've done in the past, and anonymized with @saivann's bitcoinstats script. I will write a script to determine the index page's bounce rate: how many unique visitors open it as the first page they visit but don't continue on to visit another page.
When we have enough data---I suggest 14 days, 7 days of the control text and 7 days of the test text---Saïvann and I will attempt to extract a fair sample of visits from the logs with an equal number of control and test visits, and then we'll run the bounce detection script to see what the difference between the two is.
Results and copies of all scripts used will be posted to this PR. (If you're logged into to GitHub, you can use the Subscribe button on the right side to receive notification of all subsequent comments posted on this PR.)
Bullet Text
Here is the control text, the same text currently used on Bitcoin.org for over a year:
Here is the test text. I revised/wrote it to work with the same icons, minimizing differences for the test and simplifying updating the translations if we later decide to make the test text the new control text:
Note: Bitpay's home page says, "Trusted by Over 60,000 Businesses and Organizations", and Coinbase's merchant page says, "Trusted by over 40,000 Businesses", so I think a claim of 'thousands of merchants' is warranted.
Testing The Test
[1] Before installing the cronjob, a small update to the site build script is required. I've already written it and will be submitting it in a separate PR. Also, reminder to myself: put the cronjob on a 30 second delay to avoid interfering with the regular build script.