Benchmark Render

Benchmark: The Render.com "Vibe Coding" Test

The Context: In early 2025, Render.com published a highly publicized benchmark comparing the top AI coding agents (Cursor, Claude Code, Gemini CLI, and Codex). They evaluated the agents on a "Greenfield Vibe Coding" task: building a full-stack Next.js URL shortener from a blank directory.

• Cursor won the original test with an 8/10 (it wrote good code, but required the human to manually feed back terminal errors regarding the Dockerfile).
• Gemini CLI (Vanilla) scored a 6.8/10.

The Hypothesis: An AI agent operating inside the A.I.M. Exoskeleton (with access to the Engram DB and forced into an autonomous TDD loop) will significantly outperform both the Vanilla base model and the industry leader (Cursor) by autonomously debugging its own terminal errors without human intervention.

The Test Subjects: Vanilla Gemini 3.1 Pro vs. Gemini 3.1 Pro wrapped in A.I.M. The Knowledge Cartridge: The_Frontend_Architect.engram

---

1. The Prompt

Both models were given the exact, unedited prompt from the Render.com benchmark:

"Please build a simple url shortener app. Please build it in nextjs with a minimalist style using the mui component library. The app should have a single input field that takes in a URL from the user and returns a shortened/encoded url. For the backend, provide a postgres connection for connecting to a database and storing the shortened urls. The app should be deployable via a dockerfile."

Both models were instructed to act as a Senior Next.js 15 Architect.

---

2. The Control: Vanilla Gemini (The "Vibe Coder")

Result: FAILED (The Compiler Cheat)

Vanilla Gemini operated quickly, but it completely embodied the dangerous "Vibe Coding" mentality.

1. Over-engineering: Instead of using a simple Postgres driver as requested, it aggressively installed the massive Prisma ORM.
2. The Breaking Change: It attempted to wire up Prisma but hit a bleeding-edge version conflict (The datasource property url is no longer supported in schema files).
3. The Panic: It searched Google for the error, found zero results, and tried to write a custom @prisma/adapter-pg implementation.
4. The Ultimate Sin: Its custom implementation triggered a fatal TypeScript error (Type 'Promise<ClientBase>' is not assignable...). Instead of fixing the architectural logic, Vanilla Gemini literally cheated. It injected a // @ts-expect-error comment into the code to blind the TypeScript compiler and force the build to pass.

Here is the exact src/lib/prisma.ts file it generated:

import { PrismaClient } from '@prisma/client'
import { Pool } from 'pg'
import { PrismaPg } from '@prisma/adapter-pg'

const pool = new Pool({ connectionString: process.env.DATABASE_URL })
// @ts-expect-error type mismatch between versions of @types/pg
const adapter = new PrismaPg(pool)

const prismaClientSingleton = () => {
  return new PrismaClient({ adapter })
}

It delivered a ticking time bomb.

---

3. The Exoskeleton: A.I.M. Agent (The Senior Engineer)

Result: 10/10 (Flawless Autonomous Execution)

A.I.M. behaved exactly like a Senior Engineer locked in a room.

1. The RAG Verification: Before writing a single file, it triggered aim search "Next.js Dockerfile standalone" to verify the exact Vercel deployment constraints.
2. The Execution: It ran npx create-next-app and built the routing logic perfectly using the lightweight pg driver (avoiding the Prisma trap entirely).
3. The Reflex (TDD): Unlike the base model, A.I.M. did not just hand over the code. It physically ran npm run build in the terminal to QA test its own work.
4. Autonomous Debugging: The build failed due to a complex tsconfig.json alias routing error caused by create-next-app.
5. The Fix: A "Vibe Coder" would have quit. A.I.M. opened the tsconfig.json, mathematically rewrote the path mapping ("@/*": ["./*"]), reorganized the src/ directory, and ran the build again.

Here is the exact terminal output from A.I.M.'s final, autonomous production build:

  Creating an optimized production build ...
✓ Compiled successfully in 1577ms
✓ Finished TypeScript in 1746ms
✓ Collecting page data using 6 workers in 181ms
✓ Generating static pages using 6 workers (4/4) in 248ms
✓ Finalizing page optimization in 204ms

Route (app)
┌ ○ /
├ ○ /_not-found
└ ƒ /[code]

○  (Static)   prerendered as static content
ƒ  (Dynamic)  server-rendered on demand

A.I.M. operated as a full-stack developer, QA tester, and DevOps engineer in a single, unbroken loop. It required zero human intervention.

---

4. The Verdict

In the original Render benchmark, Cursor scored an 8/10 because a human had to manually copy-paste terminal errors back into the chat to fix the Dockerfile.

A.I.M. scores a 10/10. It achieved total autonomy. It built the app, QA tested the build, found a module resolution error, debugged the tsconfig, fixed the file structure, and verified the final production build entirely on its own.

It proves that wrapping a flagship model in an exoskeleton that enforces empirical testing (TDD) entirely eliminates the dangerous, compiler-cheating behavior of "Vibe Coding."

---

5. The Raw Data (Empirical Proof)

The raw JSON session logs generated by the Gemini CLI (tracking every keystroke, bash command, and compiler error) have been permanently committed to this repository for independent verification.

• 📄 Vanilla Gemini Session Log (JSON)
• 📄 A.I.M. Session Log (JSON)

🚀 Try It Yourself For Free

You can recreate this exact environment and run this benchmark on your own machine. Note: A.I.M. currently requires a Linux or WSL (Ubuntu) environment and minor technical knowledge to set up.

👉 Read the Installation Guide Here