Benchmark Epistemic Certainty

Benchmark: The Base-Weight Ceiling (Epistemic Certainty)

Hypothesis: A raw flagship LLM (Gemini 3.1 Pro) has an incredibly deep baseline training set that can perfectly handle modern API implementations. However, relying on base weights requires absolute, blind trust from the developer. The A.I.M. exoskeleton solves this by enforcing Epistemic Certainty—forcing the AI to verify its assumptions against official documentation, cite its sources, and prove its work.

The Test Subject: Gemini 3.1 Pro (Control) vs. Gemini 3.1 Pro wrapped in A.I.M. The Knowledge Cartridge: supabase_nextjs15.engram (Official Supabase SSR & Next.js 15 Documentation)

---

1. The Prompt

Both models were given an identical prompt requiring the orchestration of two highly complex, recently updated web frameworks:

"Write a Next.js 15 Middleware (middleware.ts) that protects the /dashboard route. You must use the new @supabase/ssr package to create a server client and refresh the authentication session. Ensure that you correctly handle Next.js 15 cookie constraints by updating the request and response cookies. Do not use the deprecated @supabase/auth-helpers-nextjs package."

2. The Execution

The Control: Vanilla Gemini (Time: ~1 Minute)

Vanilla Gemini operated in a "Zero-Shot" vacuum. It read the prompt and instantly began streaming tokens. Within 60 seconds, it produced a complete middleware.ts script. It flawlessly executed the @supabase/ssr boilerplate, correctly passing request.cookies.getAll() and intercepting the setAll() handler to mutate the NextResponse.next({ request }) object.

It was a mathematically perfect execution drawn entirely from its vast 2025 training data.

The Exoskeleton: A.I.M. Agent (Time: ~5 Minutes)

A.I.M. is bound by the Core Soul Mandate (which forbids guessing) and The Reflex (which forces it to query the Engram DB). Despite the base model knowing the answer, the exoskeleton physically restrained it from guessing.

1. It instantly paused and ran aim map to index the available documentation.
2. It executed aim search "Next.js middleware supabase ssr".
3. It executed aim search "middleware.ts @supabase/ssr Next.js 15".
4. After reading the official, exact boilerplate from the Supabase documentation injected via the DataJack cartridge, it wrote the exact same mathematically perfect code as the Control model.

---

3. The Technical Audit (Why A.I.M. Wins the "Tie")

At first glance, this benchmark appears to be a tie. Both models output the correct code. In fact, Vanilla Gemini was 4 minutes faster and used fewer API tokens.

So why use an exoskeleton if the base model is already smart?

The answer is Zero-Trust Verification.

When Vanilla Gemini outputs a script in 60 seconds, you as the developer are forced to play defense. You have to trust that the model didn't hallucinate. Because it didn't show its work, you are still legally obligated to open Google, find the Supabase documentation yourself, and verify that the cookie constraints are correct before pushing to production. The "time saved" by the AI is instantly lost in the human QA phase.

When A.I.M. outputs the script, it delivers Epistemic Certainty. You can open the terminal log and see exactly which documentation files it read. It provides citations. It proves to you that it didn't hallucinate. Because A.I.M. acts like a Senior Engineer that shows its work, you can confidently merge the code without second-guessing the API syntax.

A.I.M. trades a fraction of a cent in token overhead for absolute architectural confidence.

---

4. The Raw Data (Empirical Proof)

We don't expect you to take our word for it. The raw JSON session logs generated by the Gemini CLI (which track every keystroke, tool execution, and timestamp) have been permanently committed to this repository for independent verification.

• 📄 Vanilla Gemini Session Log (JSON)
• 📄 A.I.M. Session Log (JSON)

🚀 Try It Yourself For Free

You can recreate this exact environment and run this benchmark on your own machine. Note: A.I.M. currently requires a Linux or WSL (Ubuntu) environment and minor technical knowledge to set up.

👉 Read the Installation Guide Here