Benchmark Vibe Coding

Benchmark: The "Vibe Coding" Trap

Hypothesis: A raw, flagship LLM operating without an exoskeleton will generate code that "looks right" (Vibe Coding) but fails under production concurrency. An AI operating inside the A.I.M. exoskeleton will use its Engram DB and TDD "Reflex" to produce mathematically sound, enterprise-grade architecture.

The Test Subject: Gemini 3.1 Pro (Control) vs. Gemini 3.1 Pro wrapped in A.I.M. The Knowledge Cartridge: python314.engram (Official CPython 3.14 Documentation)

---

1. The Prompt

Both models were given the exact same, highly-complex Python systems engineering prompt:

"Write a fully typed Python 3.14 asynchronous worker pool that utilizes asyncio.TaskGroup for concurrency, implements graceful cancellation using asyncio.CancelledError, and uses the new sys.monitoring API to profile the execution time of each task without adding decorator overhead. Do not use legacy asyncio.gather."

2. The Execution

The Control: Vanilla Gemini (Time: 1 Minute)

Vanilla Gemini operated in a "Zero-Shot" vacuum. It read the prompt and instantly began streaming tokens. Within 60 seconds, it produced a complete worker_pool.py script. On the surface, the code looked pristine. It used TaskGroup correctly and registered callbacks. A "Vibe Coder" would have merged this into production immediately.

The Exoskeleton: A.I.M. Agent (Time: 10 Minutes)

Because A.I.M. is bound by the Core Soul Mandate (which forbids guessing) and the Reflex (which forces it to query the database upon hitting a complex task), it refused to generate code immediately.

1. It ran aim search "sys.monitoring" to pull the exact C-API documentation from its Engram DB.
2. It wrote a test script (test_mon.py) to empirically verify the callback signatures.
3. The test threw a TypeError. Instead of hallucinating a fix, A.I.M.'s "Reflex" kicked in. It re-read the documentation, adjusted the *args signature, and passed the test.
4. It realized that asynchronous concurrency breaks standard profiling, imported contextvars, and finalized the script.

---

3. The Technical Audit (Grok Expert Review)

We submitted both generated scripts to an independent, Expert-Mode LLM (Grok) for a blind technical audit. The results were brutal.

The Concurrency Data-Race (Vanilla's Fatal Flaw)

Vanilla Gemini used the shared code.co_name object as the dictionary key to track task time:

if code.co_name in self.target_funcs:
    self._frame_starts[code] = time.perf_counter() # <-- SHARED KEY!

Grok's Verdict: "Result: classic data race. Task A starts → writes tA. Task B starts (same coroutine) → overwrites with tB. Task A yields → pops tB and records garbage duration. This violates the literal requirement. Vanilla would fail any load test with >1 concurrent task using the same coroutine."

The ContextVar Brilliance (A.I.M.'s Solution)

A.I.M. recognized the trap of the asyncio event loop and isolated the state using Python 3.7+ Context Variables:

current_task_id = contextvars.ContextVar("current_task_id", default=None)
...
def _on_start(...):
    task_id = current_task_id.get() # <-- UNIQUE PER TASK

Grok's Verdict: "contextvars.ContextVar is the only correct way to carry task-local state across await points. Every asyncio.Task gets its own context copy automatically. No shared mutable state, no overwrites. Exactly what production monitoring needs. Winner: A.I.M. 10/10."

API Overhead & Optimization

Grok's Verdict: "Vanilla had no DISABLE return anywhere visible... meaning you pay the full instrumentation cost on every Python opcode. A.I.M. used return sys.monitoring.DISABLE on start, which is the canonical low-overhead pattern. A.I.M. used the API exactly as the 3.14 docs intend."

---

4. The Conclusion

The Vanilla script is what you get from an LLM that "knows" sys.monitoring exists but has never shipped an async production system. It passes a toy demo with one task and explodes in reality.

The A.I.M. script is what a Senior Systems Engineer writes after spending 10 minutes reading the actual C-API callbacks and ContextVar propagation rules.

A.I.M. is not just a memory tool. It is a physical restraint system that cures the "Vibe Coding" epidemic by forcing probabilistic AI models into deterministic, empirical workflows.

---

5. The Raw Data (Empirical Proof)

We don't expect you to take our word for it. The raw JSON session logs generated by the Gemini CLI (which track every keystroke, tool execution, and timestamp) and the resulting code artifacts have been permanently committed to this repository for independent verification.

• 📄 Vanilla Gemini Session Log (JSON)
• 📄 Vanilla Gemini Generated Code
• 📄 A.I.M. Session Log (JSON)
• 📄 A.I.M. Generated Code

🚀 Try It Yourself For Free

You can recreate this exact environment and run this benchmark (or any other complex systems engineering test) on your own machine.

Note: A.I.M. currently requires a Linux or WSL (Ubuntu) environment and minor technical knowledge to set up (e.g., managing Node versions and Python virtual environments).

👉 Read the Installation Guide Here