Vibecode Benchmark Suite

Overview

This benchmark suite measures Java build and compile+test pipeline performance for codebases of varying sizes. It generates thousands of classes programmatically to create large, diverse test scenarios.

Quick Start

# Ensure Maven is installed
sudo apt-get update && sudo apt-get install -y maven

# Generate classes (5000 classes, ~750 lines each by default)
mvn compile exec:java -Dexec.mainClass="com.vibecode.ClassGenerator"

# Build, compile, run tests
mvn clean compile test

# View results
mvn test -Dsurefire.reportNameSuffix=results

Key Commands

Generate Classes

# Default: 5000 classes with 750 lines each
mvn compile exec:java -Dexec.mainClass="com.vibecode.ClassGenerator"

# Custom count and lines
mvn compile exec:java -Dexec.mainClass="com.vibecode.ClassGenerator" -Dexec.args="1000 500"

Build Pipeline

# Clean build + compile + tests
mvn clean compile test

# Build only (skip tests)
mvn clean compile -DskipTests

# Just compile
mvn compile

Run Tests

# Run all tests
mvn test

# Run single test
mvn test -Dtest=ClassNameTest

# Run with verbose output
mvn test -X

JMH Benchmark

# Run JMH benchmark
mvn jmh:run

# Compare iterations
mvn jmh:cmp results1/results2

Project Structure

src/main/java/com/vibecode/
  ├── ClassGenerator.java       # Generates classes programmatically
  └── VibecodeBenchmark.java    # Original benchmark placeholder

src/test/java/com/vibecode/
  ├── VibecodeAppTest.java
  └── [Generated]*Test.java     # Auto-generated unit tests

benchmarks/
  └── BenchmarkBuild.java       # JMH benchmark for timing

Dependencies

• JUnit 5 (5.10.1): Unit testing
• Mockito (5.8.0): Mock validation
• JMH (1.37): Performance benchmarking
• Guava (32.1.3-jre): Utilities (Files, Charsets)

Generated Class Domains

Domain	Algorithms
NUMBER_THEORY	GCD, primes, Fibonacci, sieve
STRING_PROCESSING	Hash, encode, frequency analysis
DATA_STRUCTURE	Tree, graph, heap operations
SORTING	QuickSort, mergesort, selection
SECURITY	Derive, hash, validate primitives
GRAPH	Adjacency, traversal, pathfinding
ALGORITHM	Search, recursion, DP
STATISTICS	Mean, median, variance
PROCESSOR	List/map processing
TRANSFORMER	Value mapping, conversion

Each Class Contains

• 50+ static constant fields
• 64+ methods (for 750-line target)
• Inner classes: Node, Result
• Main() method for standalone execution
• Pure computation only (no I/O, HTTP, networking)

Unit Tests

Every generated class gets:

• Instantiation test
• Method execution test
• Edge case validation

All tests use JUnit 5 + Mockito.

Performance Reference

Classes	Lines/Class	Build Time	Test Time	Total
100	750	~5s	~15s	~20s
500	750	~15s	~45s	~60s
1000	750	~30s	~90s	~120s
5000	750	~2-5min	~5-10min	~8-12min

Tips

1. Start small: generate 100 750, test flow, then scale up
2. Time your builds: Use JMH for accurate measurement
3. Use cache: Subsequent builds are faster
4. Skip generation: Once generated, don't regenerate for builds
5. Incremental builds: Only recompile changed files
6. Batch generation: If OOM, generate in small batches
7. Skip tests for build-only timing: mvn clean compile -DskipTests

Troubleshooting

Issue	Solution
mvn: not found	Install Maven: sudo apt-get install maven
Java version error	Ensure Java 17+: java -version
Build fails	Run mvn clean first
Out of memory	Generate in smaller batches
Test failures	Run mvn test -DfailIfNoTests=false to see all errors

Configuration

Modify pom.xml for:

• Compiler args: <arg>-parameters</arg>
• JMH includes: <includes>.*/Benchmark.*</includes>
• Test timeout: testFailureIgnore=true

Run All

# Single script to do everything
bash vibecode.sh