Turing Game

Can you tell apart humans and "machines" (modern LLMs) even in a single-turn dialog (one question, one answer)?

---

This should be a fun game inspired by the Turing Test. I plan to publish all the questions, answers and guesses (with your real names pseudonymized with chemical nicknames), so don't enter any secrete/sensitive information.

Phase 1: Questions

In Phase 1 of the Turing game, you should come up with 1-3 questions, which you think could tell apart humans and machines (i.e. LLMs). For example "How many 'r' characters are in the word 'strawberry'?" The rules are:

• The question (and expected answer) should be in English (although you can ask about any other natural language).
• The answer should not require knowledge available only to us (e.g. the color of walls in S1).
• The question should not require answers longer than say two paragraphs.

You should also specify

• which LLM model,
• setup (including harness, e.g. Copilot in VS Code) and
• exact prompt you plan to use in Phase 2. The prompt will be inserted before the question. To keep it simple, you can choose just one LLM and one prompt. In other words, you need to specify the LLM and prompt even before you know the questions you will be assigned.

Phase 2: Answers

In this phase, you will answer questions created by your colleagues: providing both your own (i.e. human) answers and answers by an LLM.

Phase 3: Guesses

In this phase, you will be assessing the usefulness of answers and guessing which answers are from humans and which from LLMs.