The "research", as seen from evidence left in the submission form and web page

[User670]

Edit: #6 is a much earlier and more in-depth comment on the research.

Guess I'm gonna be the one who spoils the thing that Cary keeps referring to as "research", with double quotes. (Someone beat me to it.)

Disclaimer: All I have are evidence that Cary left behind, and a healthy dose of speculation. I don't know any inside info, and there is still a non-zero chance that my speculations are wrong.

Near the end of the submission Google form, there is a question that asks for a seemingly unrelated word, claiming it's for "research". That immediately caught my attention: this "seemingly unrelated word" can easily be a way to identify which bucket you were put into in an A/B test. So I did what I always do: right click, view source. I found some interesting JavaScript code, and after inspecting the code and experimenting with the page, what it does is apparent:

• Every browser, when loading in the page for the first time, was placed in one of four buckets. I'll explain what these four buckets are below.
• The ID of the bucket is saved in localstorage, so that when the same browser loads the page again, it knows which version of the page it will display to you. This also means if you use a private browsing tab, you will reroll your bucket.
• Based on which bucket you are in, it displays one of 4 possible versions of the page.
• A seemingly irrelevant word at the bottom will indicate which bucket you are in, without giving out too much information. The four possible words are, ["POWDER","BROCCOLI","EARTHQUAKE","EYEBROW"].

As for the 4 possible versions of the page, there are two variables mixed and matched. These are,

• two possible scenarios, "The Bank Robbery" and "The Lands of the Gemstones". They are numerically equivalent, just morally different.
• two possible genders of main characters (i.e. you and your partner), male and female.

In case you only saw one of the scenarios and are curious about the other, here is a summary:

• "The Bank Robbery": You and your partner went robbing a bank, but after you stored your loot in a secret stash, the cops arrived and sent you both to the station. What happened next is effectively a traditional prisoners' dilemma scenario which I assume we are all familiar with. View the female version or male version.
• "The Lands of the Gemstones": There are two nations, Green and Purple, each possesses an abundance of gems of the corresponding color. Green nation people don't value their green gems much (they have too much of that) and value purple gems more, and the reverse is true for Purple nation people. You and your partner (from the opposite nation) decide to trade gems by exchanging suitcases containing the gems, and you both have the option to cheat by handing over an empty suitcase. View the female version or male version.

And yes, the two scenarios link to different versions of this github repo, namely they link to the "policestation" and "gemstone" branches respectively, where there are minor wording differences reflecting the scenario.

And here comes the speculation part: the "research" is about how the scenario affect the way people code. By presenting problems that are mathematically equivalent but presented differently, significant differences in the code between the buckets might be attributed to the different presentation of the problem.

We'll just have to wait for Cary to crunch the data and eventually share the results with us. Maybe.

[redtachyon2098]

Like you have mentioned, people have mentioned this before. I'm really curious what the results might be. Maybe an obvious correlation would show up? Maybe the majority didn't read the text, or submitted the wrong word because they cleared their cookies or used another device, in which case the results will be ruined. We would have to wait and see.

[hopperelec]

Ah this is cool!

I noticed the different branches of the repo and, after submitting, I knew it would be an A/B test but my assumption was that it was going to be something like in one scenario you're trying to get as few points as possible and in the other you're trying to get as many as possible so that they'd be put against each other expecting the other person to be trying to do the same as them. I didn't think about it much as I didn't want to spoil it for myself.

At first thought, I don't see how these changes (gender and scenario) will change the way people code, but thinking about it more it's really interesting to think about what changes these could end up making. I've been in scenarios like this in my own projects where I'm coding the exact same thing for 2 different projects, but don't realize until later and then I realize how drastically different I coded the same algorithm just due to the different use-cases.

Excited to see the results!

[ThatXliner]

Like you have mentioned, people have mentioned this before. I'm really curious what the results might be. Maybe an obvious correlation would show up? Maybe the majority didn't read the text, or submitted the wrong word because they cleared their cookies or used another device, in which case the results will be ruined. We would have to wait and see.

Honestly, I don't think the change will have much affect. That's my hypothesis