The Hallucinating Detective

by Leonard Richardson
NaNoGenMo 2023

Introduction

One of the most annoying issues with large language models when used seriously--their tendency to confabulate and confidently spout nonsense--becomes a hilarious feature when used for comedic effect. For my 2023 NaNoGenMo project I wanted to create an interplay between two pieces of software: a text generator (written by me) that could reliably induce humorous hallucinations in an LLM (trained on terabytes of human-written text).

My goal was, simply, to invoke the ELIZA effect in an LLM. Models trained on human-written text implicitly assume their input was also human-written. By feeding them automatically generated text instead, I thought I could make them infer connections where none existed.

I simulated a small world full of characters who are always looking at or showing off their posessions. These characters speak in quotes from the same Project Gutenberg texts used, in part, to train LLMs. This simulationist approach is common in NaNoGenMo entries: the text is random and meaningless, but the juxtapositions look like they ought to be meaningful. Humans fall for it, so LLMs, trained on human writings, ought to react similarly.

Unfortunately, today's LLMs are trained to be annoyingly up-front and aggressive about their own limitations. The ones I tried declined to speculate about the meaning of their statistically unlikely input; or else their speculations were wishy-washy to the point of meaninglessness, or at least unfunniness.

So I changed the prompt to goad the LLM into taking action, by casting it as an authority figure in a dangerous and adversarial world. In The Hallucinating Detective, the language model is the detective in a murder mystery. The people became suspects or dead bodies; the posessions became clues. The simulated world becomes a text adventure that ignores all player input and keeps moving towards a conclusion, or at least an ending.

Some language models kept their distance from the role of detective, others embraced it, but most of them at least tried to solve the unsolvable mystery.

Table of Contents

These twenty-four detective stories, created by a variety of models, are hosted at my personal site.

1. The Mystery of the Foolish Door
   starring Roko as "Sleuth Seraphina"
2. The Encrusted Handful Of Change Affair
   starring gpt4all-falcon-q4_0
3. The Adventure of the Smothered Egg Beater
   starring replit-code-v1_5-3b-q4_0
4. The Adventure of the Workable Scotch Tape
   starring gpt4all-13b-snoozy-q4_0
5. The Mystery of the Footy Box Of Chocolates
   starring nous-hermes-llama2-13b
6. The Adventure of the Healthier Blowdryer
   starring em_german_mistral_v01
7. The Case of the Registering Coffee Pot
   starring rift-coder-v0-7b-q4_0
8. The Revelatory Tube Of Lip Balm Affair
   starring mistral-7b-instruct-v0
9. The Repellent Chocolate Affair
   starring starcoder-q4_0
10. The Adventure of the Skeleton Roll Of Gauze
    starring mistral-7b-openorca
11. The Case of the Uninspired Light
    starring Roko as "The Insightful Enigma"
12. Sherlock Holmes and The Draining Pen Affair
    starring em_german_mistral_v01
13. The Case of the Prodigious Straw
    starring wizardlm-13b-v1
14. The Adventure of the Humiliating Zipper
    starring rift-coder-v0-7b-q4_0
15. The Automatic Pants Affair
    starring gpt4all-falcon-q4_0
16. The Case of the Forehand Wedding Ring
    starring replit-code-v1_5-3b-q4_0
17. The Adventure of the Flexible Sword
    starring wizardlm-13b-v1
18. The Case of the Broadest Screw
    starring mistral-7b-openorca
19. The Case of the Neutered Keyboard
    starring wizardlm-13b-v1
20. The Case of the Priced Whale
    starring nous-hermes-llama2-13b
21. The Case of the Backstage Squirrel
    starring wizardlm-13b-v1
22. The Adventure of the Rapid-Fire Squirrel
    starring rift-coder-v0-7b-q4_0
23. The Subjugated Chain Affair
    starring em_german_mistral_v01
24. The Swarthy Dog Affair
    starring wizardlm-13b-v1

Tools used

• Original source code.
• The GPT4All collection of models.
• The Roko chatbot from the Random Number Venerators Discord server.
• The llm and llm-gpt4all Python libraries.
• My own olipy library, including the data aggregated from Darius Kazemi's corpora
• My ol' reliable, the Project Gutenberg 10K DVD.