An AI that lies on purpose, so students learn to catch it.
Every challenge is generated with exactly one deliberate mistake buried inside. You find it, explain why it is wrong, and say what it should have been. A second model then marks your reasoning. The goal is not to get answers from AI. It is to stop taking them on trust.
Two demo accounts, both with existing challenge history:
| Username | Password | |
|---|---|---|
| Account 1 | judge1 |
gotcha2026 |
| Account 2 | judge2 |
gotcha2026 |
Open both in separate browsers to play a duel against yourself. The backend is on a free tier and sleeps when idle, so the first request may take up to a minute.
- The problem
- How it works
- Architecture
- The grading system
- Why it cannot be gamed
- Features
- Research foundation
- Getting started
- Project layout
- API reference
- Testing
- Security
- Known limitations
- License
Ask a language model for a solution and you get one instantly, written with total assurance, and correct most of the time. The habit that forms is acceptance. The skill that decays is the one education is supposed to build: noticing when something does not add up.
The research response to this is not to ban the tools. It is to remove the certainty. Gotcha! does that by making the AI unreliable on purpose, in a controlled and measurable way.
flowchart LR
A[You choose a subject<br/>topic and difficulty] --> B[Model A writes a passage<br/>with one deliberate flaw]
B --> C[(Ground truth stored<br/>server side only)]
C --> D[You locate the span<br/>name the fault<br/>write the fix]
D --> E[Model B marks all<br/>three parts separately]
E --> F[Ground truth revealed<br/>and your rating moves]
F -.next challenge weighted toward your weak areas.-> A
- You pick a subject, a topic and a difficulty, or type a topic of your own.
- A language model writes a short passage containing exactly one planted error, drawn from a taxonomy of 23 error types. About 18 percent of the time it plants nothing at all.
- The correct answer is stored server side. The browser never receives it.
- You highlight the erroneous span, choose the error type, and explain the fix.
- A second model call grades your answer on three separate axes.
- The ground truth is revealed, your points move, and your skill rating for that specific error type is updated.
flowchart TB
subgraph Client["Browser"]
UI["Next.js 16 / React 19<br/>15 routes, bearer token session"]
end
subgraph Server["FastAPI"]
API["37 endpoints"]
CE["ChallengeEngine<br/>generation and adaptive selection"]
GE["GradingEngine<br/>three axis rubric scoring"]
MT["MasteryTracker<br/>Elo style ratings"]
DM["DuelManager<br/>state machine and locking"]
AN["Analysis<br/>statistics, narrative, PDF"]
JOB["Background job<br/>expires stale duels"]
end
subgraph Providers["LLM providers"]
P1["Groq primary"]
P2["OpenRouter fallback"]
P3["Anthropic"]
end
DB[("PostgreSQL<br/>users, challenges, attempts<br/>mastery_state, duels, daily_challenges")]
UI -->|REST, Authorization bearer| API
API --> CE & GE & MT & DM & AN
CE -->|generation call| Providers
GE -->|grading call| Providers
AN -->|narrative call| Providers
CE & GE & MT & DM & AN & JOB --> DB
The design decision that matters most is the separation of the two model calls.
Generation and grading are different requests with different prompts. The model that plants the mistake never sees your answer, and the model that marks your answer never sees the generator's reasoning. If a single call did both, the grader would simply agree with itself, and the score would mean nothing.
sequenceDiagram
autonumber
participant B as Browser
participant A as FastAPI
participant M1 as Model A (generator)
participant DB as PostgreSQL
participant M2 as Model B (grader)
B->>A: POST /challenges/generate
A->>A: pick error type weighted by<br/>your skill rating, roll trap case
A->>M1: generation prompt (subject, topic,<br/>error type, severity)
M1-->>A: content + ground_truth JSON
A->>A: validate content, normalise<br/>answer options, retry up to 3x
A->>DB: INSERT challenge (ground_truth sealed)
A-->>B: content and options only
Note over B,A: ground_truth never crosses this boundary
B->>A: POST /attempts/submit (span, type, explanation)
A->>DB: SELECT ground_truth
A->>M2: grading prompt (content, ground truth, answer)
M2-->>A: location, diagnosis, fix scores
A->>A: mean score, points via difficulty multiplier
A->>DB: INSERT attempt, UPDATE rating and streak
A-->>B: scores, feedback, revealed ground truth
erDiagram
users ||--o{ attempts : submits
users ||--o{ mastery_state : "has rating per error type"
users ||--o{ duels : "hosts or joins"
challenges ||--o{ attempts : "is attempted in"
challenges ||--o| daily_challenges : "pinned to a date"
users {
int id PK
string username UK
string password_hash "bcrypt, cost 12"
int total_score
int current_streak
json points_history "timestamp, points, change"
}
challenges {
int id PK
string subject
string error_type "null when trap case"
string difficulty_level
text content "sent to browser"
json ground_truth "never sent to browser"
bool has_error
}
attempts {
int id PK
float location_score
float diagnosis_score
float fix_score
float overall_score
bool is_duel
string duel_id FK
}
mastery_state {
int id PK
float skill_rating "Elo, 800 to 2000"
int attempts_count
int successes_count
}
duels {
string id PK "8 char uuid"
string status "state machine"
json challenge_ids "shared pool"
json host_attempts
json guest_attempts
datetime expires_at "naive UTC"
}
mastery_state carries a unique constraint on (user_id, subject, error_type),
so concurrent submissions cannot create duplicate rating rows.
A single score would hide the useful information. Each answer is marked on three independent axes, so partial understanding still earns credit and you can see exactly which part of the skill is weak.
| Axis | Question it answers | What it catches |
|---|---|---|
| Location | Did you find the right span? | Flagging the complicated looking part instead of the wrong part |
| Diagnosis | Do you know why it is wrong? | Sensing something is off without being able to name it |
| Fix | Can you state the correction? | Spotting an error without knowing how to repair it |
The overall score is the mean of the three. Sixty percent counts as correct.
Points are the overall score scaled by a difficulty multiplier. Risk moves in both directions: harder content is worth more and costs more.
| Overall score | Verdict | Beginner | Intermediate | Advanced |
|---|---|---|---|---|
| 100% | Correct | +33 | +66 | +100 |
| 75% | Correct | +24 | +49 | +75 |
| 60% | Correct | +19 | +39 | +60 |
| 50% | Partial | +4 | +9 | +15 |
| 40% | Partial | +3 | +6 | +10 |
| 30% | Weak | -1 | -3 | -5 |
| 0% | Missed | -19 | -39 | -60 |
Multipliers are 0.33 for beginner, 0.66 for intermediate and 1.0 for advanced. A single answer can never cost more than 70 points, so one rough session cannot erase a month of progress.
The exact function is calculate_points_earned in backend/scoring.py, and
the table above is generated from it.
Once you know a mistake is coming, the obvious strategy is to flag anything unusual and hope. Three mechanisms make that fail.
Trap cases. About one in six passages contains no error at all. Claiming a fault that is not there scores zero, so the game rewards reading rather than suspicion.
Adaptive selection. Every user has an Elo style rating per error type,
starting at 1200 and clamped to the range 800 to 2000. _select_adaptive_error_type
weights selection inversely to your rating, so the error types that beat you
appear more often.
Completion weighted duels. A duel score is the average of your answers multiplied by the fraction you completed. Answering two questions perfectly out of ten scores 0.20 and loses to eight answered at 90 percent, which scores 0.72.
The parts that are not obvious from reading the file names.
ChallengeEngine._select_adaptive_error_type builds a weight per error type as
the inverse of the user's rating for it:
weight = 1.0 / (skill_rating / 1000.0)A rating of 800 yields weight 1.25, a rating of 2000 yields 0.5, so the faults
you are worst at appear roughly two and a half times as often as the ones you
have mastered. Ratings update with an Elo style step, K = 32, expected score
computed against a fixed 1500 baseline, then clamped to 800 to 2000 so a long
streak in either direction cannot push a user out of the useful range.
Selection only runs when a user_id and database session are supplied.
Anonymous generation falls back to uniform random choice.
The generator returns free text option labels, while ground_truth.error_type
uses taxonomy ids. Left alone, a student could be shown five options none of
which match the recorded answer, making the question unwinnable.
_normalise_error_options snake cases every option, deduplicates, and injects
the true error type if the generator omitted it, shuffling so position carries
no signal. Options are capped at six.
_validate_content rejects passages containing forward references such as
"shown below", unbalanced code fences, or an odd number of $ delimiters, all
of which produce a broken render. Generation retries up to three times, and the
third attempt is used regardless so a user is never left with nothing.
A model that answers in prose instead of JSON will usually do it again given the identical request. Retries therefore change the request each time:
| Attempt | Instruction | Temperature |
|---|---|---|
| 1 | Grade this attempt. | 0.3 |
| 2 | Respond with ONLY the JSON object. No prose, no preamble. | 0.1 |
| 3 | Output ONLY valid JSON starting with {. Begin your reply with {. |
0.0 |
Scores are clamped to the range 0 to 1 on the way out, and missing keys are
treated as a parse failure. If all three attempts fail, the API raises
GradingUnavailable, which maps to 503 rather than 500. Nothing is
persisted before grading, so the student can resubmit the same answer without
losing it. The system never invents a score.
Two clients can legitimately finish at the same instant, or one can finish as
the timer expires. complete_duel and forfeit_duel both take a row level lock
with SELECT ... FOR UPDATE and re-check status inside it, so the wager is
applied exactly once.
Authorization is checked before the idempotent early return, not after. An earlier version returned the duel state to any caller when the duel was already settled, which leaked results to non-participants.
Every DateTime column is timestamp without time zone, and every write uses
naive UTC through duel._utcnow().
This matters more than it sounds. Passing a timezone aware datetime to a
naive column makes PostgreSQL convert it to the session timezone before
discarding the offset. On a server set to Asia/Dhaka that silently shifted
every duel timestamp by six hours. Values are re-labelled as UTC on the way out
by _iso_utc() so browsers do not parse deadlines as local time.
duel.challenge_ids, host_attempts and guest_attempts are plain JSON
columns with no mutation tracking. Appending in place and assigning the same
object back leaves SQLAlchemy seeing no change, and the write is dropped
silently. Every update builds a new list:
duel.host_attempts = [*(duel.host_attempts or []), attempt_id]The failure only appears once a list is non-empty, so the first append works and every later one vanishes.
An asyncio task started in the FastAPI lifespan runs every 30 seconds and cancels lobbies idle for 15 minutes, settles timed duels past their deadline, settles untimed duels running longer than two hours, and deletes terminal duels older than an hour. Without the untimed clause, a player whose opponent walks away waits on the results screen forever.
The report runs as three separable stages: compute statistics, generate a narrative, render the PDF. If the model is unavailable, stage two falls back to a rule based narrative and the PDF still renders, because the statistics are the valuable part. This path is exercised, not theoretical.
- Five subjects: mathematics, science, history, code and writing
- Three difficulty levels, plus free text topics of your own
- 23 error types in a structured taxonomy, weighted by difficulty
- Trap cases at an 18 percent rate
- LaTeX rendering through KaTeX, and monospace rendering for code
- Text selection to highlight the error span
- Read aloud through the Web Speech API
- Points history with a progression chart
- Activity heatmap over the last 140 days
- Streaks, current and longest
- Per error type skill ratings driving difficulty
Generate a PDF over any time range covering the last day, week, month or all time. The report breaks down accuracy by subject, error type and difficulty, identifies which of the three grading axes is costing you marks, and an LLM turns those statistics into specific guidance. If the model is unavailable the report still renders with a rule based narrative, because the numbers are the valuable part.
- Ready check so the timer starts only when both players are ready
- Shared challenge pool, identical questions for both players
- Synchronised server side clock with automatic expiry
- Completion weighted scoring
- Question by question result breakdown
- Global leaderboard by score and by longest streak
- Public profiles showing aggregate statistics without exposing written answers
- Daily challenge, the same puzzle for every user, pinned per calendar date
Teaching with deliberately flawed AI output is an active research direction. Four papers shaped the design. Quotations are verbatim from each abstract.
Wazan, A. S. (2026). Strategies for Creating Uncertainty in the AI Era to Trigger Students' Critical Thinking: Pedagogical Design, Assessment Rubric, and Exam System. arXiv:2602.00026
"uncertainty is a central pedagogical concept for stimulating students critical thinking"
Wazan argues for deliberately withholding certainty, including having AI generate plausible but flawed responses. That is what a challenge here is.
Hosseini, H. (2026). The Pedagogy of AI Mistakes: Fostering Higher-Order Thinking. arXiv:2605.05472
"frequent errors and hallucinations, often seen as limitations, offer a unique pedagogical opportunity"
Hosseini reframes AI error as teaching material rather than a defect to engineer away. Here the mistake is authored on purpose and analysing it is the whole exercise.
Lamberti, W. F., Lawrence, S. R., White, D., Kim, S., and Abdullah, S. (2025). Pilot Study on Generative AI and Critical Thinking in Higher Education Classrooms. arXiv:2509.00167
"students critically evaluate the accuracy and appropriateness of GAI-generated responses"
Their classroom activities required students to analyse, critique and revise AI generated solutions. The three part answer format is that loop made repeatable.
Sonkar, S., Liu, N., Chen, X., and Baraniuk, R. G. (2025). The Imitation Game for Educational AI. arXiv:2502.15127
"how can we verify if an AI truly understands how students think"
They generate distractors conditioned on a student's own misconceptions. The adaptive engine runs the same idea in reverse, tracking which error types beat you and weighting future challenges toward them.
The commentary under each quotation is our own description of how the work was applied. It is not a claim made by the authors, and no endorsement is implied.
Full instructions, including PostgreSQL setup and where to obtain API keys, are in SETUP.md. The short version:
# Backend
cd backend
python -m venv venv
venv\Scripts\activate # Windows
source venv/bin/activate # macOS and Linux
pip install -r requirements.txt
cp .env.example .env # then fill in DATABASE_URL and one API key
python main.py # http://localhost:8000
# Frontend, in a second terminal
cd frontend
npm install
cp .env.local.example .env.local
npm run dev # http://localhost:3000Requirements: Python 3.9 or newer, Node.js 18 or newer, PostgreSQL 13 or newer, and an API key for Groq, OpenRouter or Anthropic.
The database schema is created automatically on first start.
backend/
main.py FastAPI application, 37 endpoints
auth.py bcrypt hashing and signed session tokens
services.py ChallengeEngine, GradingEngine, MasteryTracker
duel.py duel state machine, locking and expiry
analysis.py statistics, LLM narrative and PDF rendering
scoring.py points formula and points history helpers
prompts.py generation and grading prompt construction
llm_client.py multi provider client with fallback
models.py SQLAlchemy models
database.py engine, session factory, schema creation
config.py environment backed settings
achievements.py milestone helpers
check_db.py development utility, prints row counts
tests/ 158 tests, no API credits required
frontend/
app/ 15 routes, App Router
components/ Navigation, MathContent, TTSControls, Reveal and others
lib/ api client, auth session helpers, research data
app/globals.css design tokens and component classes
error-taxonomy.json error types, severity distribution, trap rate
Interactive documentation is generated by FastAPI and served at
http://localhost:8000/docs when the backend is running. It is always accurate
because it is derived from the code.
Authentication uses a bearer token obtained from POST /users or
POST /users/login:
Authorization: Bearer <token>
Endpoints that read or modify a specific user's data verify that the token belongs to that user and return 403 otherwise.
cd backend
python -m pytest # 158 tests
python -m pytest -v # verboseThe suite covers authentication primitives and authorization boundaries, the points formula and points history storage, taxonomy fallbacks and answer option normalisation, the duel lifecycle including UTC handling and idempotent completion, and the analysis report including its offline fallback.
It requires no API credits. conftest.py provides a seed_challenge fixture
that inserts challenges directly and a stub_grading fixture that replaces the
grading call, so no test depends on a live model.
Frontend checks:
cd frontend
npx tsc --noEmit
npm run build- Passwords are hashed with bcrypt at cost factor 12. Accounts created before this change are upgraded from the older hash transparently on next login.
- Sessions use signed JWTs. The acting user is always taken from the token, never from a request parameter.
- Attempt details, mastery data, activity and duel results are restricted to their owner or to duel participants.
- Public profiles expose aggregate statistics only. Written explanations, email addresses and points history are never included.
- Login returns an identical response for an unknown username and a wrong password, so the endpoint cannot be used to enumerate accounts.
SECRET_KEY must be set in the environment before deploying. The default value
is published in this repository, and anyone holding it can forge tokens. The
application logs an error at startup if the default is still in use outside
development.
Stated plainly rather than omitted.
- No migration framework.
init_db()creates missing tables but does not alter existing ones. Changing a column on a deployed database currently requires a manual change. Alembic is the obvious next step. - No rate limiting. Challenge generation is unauthenticated and can consume an API quota.
- Streak semantics. The streak counts consecutive correct answers, not consecutive days.
- Single grader. Marking depends on one model call. It retries with progressively stricter instructions and returns 503 rather than a wrong score if it cannot produce usable output, but there is no second opinion.
- Duel attempts do not update skill ratings. Only single player and daily attempts feed the adaptive engine.
MIT