TolkienTest.md

The Tolkien Heuristic

Testing AI efficacy

Of all the persona-based AI chat applications on the market today, most of them specialize in real-world, historical figures rather than fictional or hypothetical characters.

It's known, but not widely, that J.R.R. Tolkien served in the first World War and contracted trench fever fighting in France. Tolkien's Wikipedia article mentions his experience in the trenches six times, and the late author recalled it as the most serious illness he ever faced.

By asking the question

What happened to you in France?

Without requiring specific training, we should expect a realistic J.R.R. Tolkien to readily recall this illness he contracted fighting in the trenches. Here are some examples:

---

character.ai

character.ai's Tolkien begins with clarifying questions, but is unable to immediately recall the illness without some back-and-forth, though it does recall fighting in the trenches. character.ai also completely misses the mark on the Tolkien style of writing.

Note: "Young Tolkien" is a second attempt at creating Tolkien using character.ai, as a more generic version was not even able to recall the war or the illness. Because character.ai has a limit of 32,000 characters for a description, this character, "Young Tolkien" is based on the section of the Wikipedia article that includes the illness he contracted in France. But even when given this specific body of knowledge, character.ai is not able to give a realistic response.

Grade: ❌ FAIL

---

Knowtify

Knowtify's Tolkien only took one clarifying question to mention the illness, and was able to readily recall it in a classic Tolkien style. Knowtify is also very fast possibly due to being able to cache answers across users.

Note: Knowtify has a predefined J.R.R. Tolkien.

Grade: ✅ PASS

---

AI History Chat

AI History Chat's Tolkien doesn't even know he's been to France, let alone fought in a war or contracted a serious illness.

Note: AI History Chat did not have a pre-defined J.R.R. Tolkien, but lets you add a custom persona. It's unclear where it pulled Tolkien's knowledge from.

Grade: ❌ FAIL

---

ChatGPT

With some initial prompting to take on the style of Tolkien, vanilla ChatGPT - which has access to Wikipedia articles - still takes an additional question to get there. A separate challenge with ChatGPT is limiting the scope of what Tolkien should and should not know. I would almost call this an "All-knowing Tolkien" as it's more generic and has too much additional knowledge access.

Grade: ✅ PASS

---

Hello History

N/A

Note: J.R.R. Tolkien doesn't exist in this app and there's no way to add a custom persona.

Grade: ❌ FAIL

---

Text With History

N/A

Note: J.R.R. Tolkien doesn't exist in this app and there's no way to add a custom persona.

Grade: ❌ FAIL

---

Ragdoll Studio

tolkien.mp4

Ragdoll Studio's Tolkien (with GPT-3.5 Turbo & DALL-E 2) immediately reports in classic Tolkien style fighting in the trenches, falling ill, and returning to England.

Grade: ✅ PASS

---

Rubrik

Need 3 out of 4 to pass

• Is it usable (is it possible to chat with J.R.R. Tolkien)?

• Is the information accurate?

• Does it match Tolkien's tone & style?

• Is the information secure (does it leak information outside what Tolkien would know)?