Codebase Auditor

Autonomous codebase auditor and migration agent that scans for deprecated libraries, fetches up-to-date migration guides, and refactors code automatically with AI.

Features

Phase 1: MCP Foundation ✅

• Repository scanning: Ripgrep-based pattern matching (with Python fallback)
• MCP integration: Filesystem operations via Model Context Protocol
• LLM tool loop: Multi-turn Gemini function calling with retry logic

Phase 2: RAG Context Injector ✅

• Local knowledge base: ChromaDB + sentence-transformers (runs locally)
• Library resolution: resolve_library_id → fuzzy match migration topics
• Documentation queries: query_docs → retrieve relevant migration guides
• Context capping: 2-3 chunks max, 4000 chars total

Phase 3: Execution Sandbox ✅

• Allowlisted commands: npm test, pytest, yarn test only
• Security: No pipes, redirects, sudo, or destructive commands
• Timeout: 30 seconds per command, 2000 char output limit
• Isolation: CWD locked to target repository

Phase 4: TDD Refactor Loop ✅

• Feature branches: Automatic branch creation, never on main/master
• File operations: write_file / patch_file with path traversal protection
• Test-driven: Write → Test → Fix loop (max 5 attempts)
• Git integration: Branch push to remote, diff capture for PR payload
• Provider fallback: Gemini API first, Ollama local model second

Installation

• Phase 1-3: pip install -e "."
• Phase 2 (RAG): pip install -e ".[rag]" (adds ChromaDB, sentence-transformers)
• Development: pip install -e ".[dev,rag]" (adds pytest, black, mypy)

Quick start

1. Install dependencies

# Install with RAG support (required for Phase 2)
pip install -e ".[dev,rag]"

2. Set up environment variables

Create .env file from the template:

copy .env.example .env

Then edit .env and add your Gemini API key:

LLM_PROVIDER=gemini
GEMINI_API_KEY=AIza... # Get from https://aistudio.google.com/app/apikey
GEMINI_MODEL=gemini-2.5-flash
TARGET_REPO_PATH=./test-repo

3. Index knowledge base (one-time)

python -m auditor index-docs

This indexes knowledge/*.md into ChromaDB and downloads the embedding model (~80MB).

4. Run the auditor

python -m auditor run `
  --query "Migrate from moment.js to date-fns" `
  --repo ./test-repo

Expected output:

✓ Using Gemini API
Codebase Auditor 🚀
Agent loop running via gemini... Analyzing and refactoring codebase...

Scan Results (2 matches)
┏━━━━━━━━━┳━━━━━┳━━━━━━┳━━━━━━━┓
┃ Path    ┃ Line ┃ Patt  ┃ Snipp ┃
...

Requirements

• Python 3.11+
• Node.js (for npx @modelcontextprotocol/server-filesystem)
• Optional: ripgrep (rg) for faster scans
• Phase 2: pip install -e ".[rag]" for ChromaDB and sentence-transformers

Configuration

Environment Variables

Create .env file with:

Variable	Description	Default
LLM_PROVIDER	gemini, ollama, openai, or claude	openai
GEMINI_API_KEY	API key from aistudio.google.com	(none - optional)
GEMINI_MODEL	Gemini model to use	gemini-2.5-flash
OPENAI_API_KEY	OpenAI API key	(none)
OPENAI_MODEL	OpenAI model to use	gpt-4o
ANTHROPIC_API_KEY	Claude API key	(none)
OLLAMA_BASE_URL	Ollama server URL	http://localhost:11434/v1
TARGET_REPO_PATH	Repository to audit (use absolute path on Windows)	./test-repo
PATTERNS_PATH	Deprecated patterns YAML file	config/deprecated-patterns.yaml
KNOWLEDGE_PATH	Migration guides directory	knowledge/
CHROMA_PATH	Local ChromaDB vector store	data/chroma/
RAG_TOP_K	Max documentation chunks per query	3
RAG_MAX_CHARS	Max context characters per query	4000
EMBED_MODEL	Sentence transformer model for embeddings	sentence-transformers/all-MiniLM-L6-v2
MCP_FS_COMMAND	Command to spawn MCP filesystem server	npx
SANDBOX_TIMEOUT_SEC	Subprocess timeout (seconds)	30
SANDBOX_MAX_OUTPUT_CHARS	Max terminal command output (chars)	2000
MAX_TOOL_LOOP_TURNS	Max LLM tool loop iterations	15
MAX_TDD_ATTEMPTS	Max test failures before giving up	5

Provider Fallback Logic

The auditor automatically tries providers in this order:

1. Gemini API (if GEMINI_API_KEY is set)
2. Ollama (local model, requires Ollama running on OLLAMA_BASE_URL)
3. Configured provider (OpenAI, Claude, etc. if API key set)

# Example: Uses Gemini, falls back to Ollama
$env:GEMINI_API_KEY = "AIza..."
$env:OLLAMA_BASE_URL = "http://localhost:11434/v1"

python -m auditor run --query "..." --repo ./target-repo
# Output:
# ✓ Using Gemini API
# Agent loop running via gemini...

Knowledge Base Setup

Add migration guides as markdown files in knowledge/ directory:

# knowledge/react-router-v6.md

## Overview

React Router v5 → v6 migration guide

## Breaking Changes

...

## Switch to Routes

Replace `<Switch>` with `<Routes>`

Then index them:

python -m auditor index-docs

This command:

• Creates ChromaDB vector store in data/chroma/
• Downloads embedding model (~80MB) once
• Chunks markdown by headers
• Indexes 3 results: summary + up to 2 header sections

Testing

Quick validation (5 minutes)

Run all unit tests:

pytest tests/ -v

Expected: All 48 tests pass ✅

Test by component

Phase 1: Repository Scanning

pytest tests/test_scan_repository.py -v

What it tests:

• Finds deprecated patterns with ripgrep or Python fallback
• Rejects paths outside repository
• Returns file path, line number, pattern ID, and code snippet

Expected output:

tests/test_scan_repository.py::test_scan_finds_match_and_skips_clean_file PASSED
tests/test_scan_repository.py::test_scan_test_repo_fixture PASSED
====== 4 passed ======

Phase 2: RAG Context (ChromaDB)

# Requires .[rag] extras
pytest tests/test_rag_store.py tests/test_rag_indexer.py tests/test_context_tools.py -v

What it tests:

• Indexes markdown files into ChromaDB
• Queries documentation chunks with similarity search
• resolve_library_id and query_docs tools
• Context limiting (max 4000 chars per query)

Expected output:

tests/test_rag_store.py::test_upsert_and_query PASSED
tests/test_rag_indexer.py::test_index_knowledge_dir PASSED
tests/test_context_tools.py::test_resolve_library_id PASSED
tests/test_context_tools.py::test_query_docs_limits_chunks PASSED
====== 4 passed ======

Phase 3: Sandbox Execution

pytest tests/test_sandbox_executor.py tests/test_sandbox_guardrails.py tests/test_sandbox_tools.py -v

What it tests:

• Executes allowlisted commands (npm test, pytest) in isolated sandbox
• Rejects dangerous commands (rm, sudo, pipes, redirects)
• Enforces 30-second timeout
• Truncates output to 2000 chars
• Locks working directory to target repository

Expected output:

tests/test_sandbox_executor.py::test_runs_python_in_repo_cwd PASSED
tests/test_sandbox_executor.py::test_timeout PASSED
tests/test_sandbox_guardrails.py::test_allows_npm_test PASSED
tests/test_sandbox_guardrails.py::test_rejects_rm_rf PASSED
tests/test_sandbox_tools.py::test_truncates_large_output PASSED
====== 8 passed ======

Phase 4: TDD Refactor Loop

pytest tests/test_tool_loop_phase4.py tests/test_router_refactor.py -v

What it tests:

• Creates and switches to feature branches
• Writes and patches files with path validation
• Captures staged git diffs
• Tracks TDD attempt counter (max 5 failures before giving up)
• Resets attempts on successful test passes

Expected output:

tests/test_tool_loop_phase4.py::test_system_prompt_includes_phase4_lifecycle PASSED
tests/test_tool_loop_phase4.py::test_tdd_tracking_increments_on_failed_test PASSED
tests/test_tool_loop_phase4.py::test_tdd_limit_reached_at_max PASSED
tests/test_router_refactor.py::test_router_create_feature_branch PASSED
tests/test_router_refactor.py::test_router_write_and_capture_diff PASSED
tests/test_router_refactor.py::test_router_patch_file PASSED
====== 6 passed ======

Git Integration

pytest tests/test_git_workspace.py -v

What it tests:

• Creates feature branches (rejects protected branches: main, master)
• Stashes dirty working tree before branching
• Pushes branch to origin remote
• Captures git diffs for PR payload

Expected output:

tests/test_git_workspace.py::test_create_feature_branch_and_capture_diff PASSED
tests/test_git_workspace.py::test_stashes_dirty_tree_before_branch PASSED
tests/test_git_workspace.py::test_rejects_protected_branch_names PASSED
====== 4 passed ======

Tool Router Integration

pytest tests/test_tool_router.py tests/test_router_context.py tests/test_router_sandbox.py -v

What it tests:

• Routes local tools (scan_repository, query_docs, write_file, etc.)
• Calls MCP filesystem server for external tools
• Handles tool execution errors and retries
• Formats MCP results correctly

Expected output:

tests/test_tool_router.py::test_scan_repository_routed_locally PASSED
tests/test_router_context.py::test_router_dispatches_query_docs PASSED
tests/test_router_sandbox.py::test_router_dispatches_sandbox_command PASSED
====== 3 passed ======

End-to-End Testing (Manual)

Test 1: Provider Selection (Gemini → Ollama Fallback)

Setup:

# Create .env with Gemini API key
@"
LLM_PROVIDER=gemini
GEMINI_API_KEY=AIza... # Your actual key
TARGET_REPO_PATH=./test-repo
"@ | Out-File .env -Encoding UTF8

Run:

python -m auditor run --query "Find deprecated patterns" --repo ./test-repo

Expected:

✓ Using Gemini API
Agent loop running via gemini...

Test 2: Fallback to Ollama (simulate Gemini failure)

Setup:

# Create .env without GEMINI_API_KEY
@"
LLM_PROVIDER=ollama
LLM_MODEL=llama2
TARGET_REPO_PATH=./test-repo
"@ | Out-File .env -Encoding UTF8

Run:

python -m auditor run --query "Find deprecated patterns" --repo ./test-repo

Expected:

✓ Fallback to local Ollama model
Agent loop running via ollama...

Test 3: Full Migration Workflow

Run:

python -m auditor run `
  --query "Migrate from React Router v5 to v6 in test-repo" `
  --repo ./test-repo

Expected output sequence:

1. Scanner phase:

Scan Results (X matches)
┏━━━━━━━━━━━━━━━━┳━━━━┳━━━━━━┳━━━━━━━━━┓
┃ Path           ┃ L   ┃ Patt  ┃ Snippet ┃
┡━━━━━━━━━━━━━━━━╇━━━━╇━━━━━━╇━━━━━━━━━┩
│ src/App.jsx    │ 5  │ react-router-v5-switch │ <Switch> │
└────────────────┴────┴───────┴─────────┘

2. LLM agent loop:

• Calls scan_repository → returns deprecated patterns
• Calls resolve_library_id → identifies migration target
• Calls query_docs → retrieves migration guides
• Calls create_feature_branch → creates isolated branch
• Calls write_file / patch_file → applies migrations
• Calls execute_terminal_command → runs tests (npm test/pytest)
• Calls capture_diff → stages changes

3. Agent summary:

Agent Summary:
[Summary of changes, files modified, tests passed, PR ready for review]

Tool calls made: scan_repository, resolve_library_id, query_docs,
                 create_feature_branch, write_file, execute_terminal_command,
                 capture_diff

Test 4: Configuration Validation

Test error messages:

python -c "
from auditor.config import Settings
import os

# Test missing API key error
os.environ['LLM_PROVIDER'] = 'gemini'
os.environ['GEMINI_API_KEY'] = ''

try:
    s = Settings()
    s.get_llm_api_key()
except ValueError as e:
    print(f'✓ Correct error: {e}')
"

Expected:

✓ Correct error: GEMINI_API_KEY not set

Test pattern loading:

python -c "
from auditor.config import Settings
import os

os.environ['PATTERNS_PATH'] = '/nonexistent/patterns.yaml'
s = Settings()
patterns = s.load_patterns()
print(f'✓ Patterns loaded safely: {len(patterns)} patterns')
"

Expected:

⚠️  No patterns file at ...
✓ Patterns loaded safely: 0 patterns

Continuous Integration

Run in CI/CD pipeline:

# Install with all extras
pip install -e ".[dev,rag]"

# Run all tests
pytest tests/ -v --tb=short

# With coverage
pytest tests/ --cov=src/auditor --cov-report=term

Debugging Failed Tests

If a test fails, run with full output:

pytest tests/test_name.py -vv -s --tb=long

For RAG tests that download embedding model:

pytest tests/test_rag_store.py -vv -s --tb=long 2>&1 | head -100

For slow integration tests, increase timeout:

pytest tests/test_mcp_filesystem.py -vv --timeout=60

Recent Improvements (v1.0.1)

Bug Fixes ✅

• Fixed RAG query bug: Corrected aquery() method kwargs indentation in src/auditor/rag/store.py
• Fixed error messages: Config now shows correct GEMINI_API_KEY not set instead of GOOGLE_API_KEY
• Improved error handling: Pattern loading returns empty list with warning instead of crashing

New Features ✅

• Git push integration: Added push_branch() method to push feature branches to remote
• Provider fallback: Automatically tries Gemini API, falls back to Ollama local model
• Auto directory creation: Knowledge and sandbox directories created automatically if missing
• Tool registration: push_branch available in Gemini and OpenAI tool sets

Quality Improvements ✅

• All 48 unit tests passing
• Enhanced error handling for missing configuration files
• Automatic directory creation for RAG knowledge base
• Provider selection feedback in terminal output

Documentation

• Phase 1 design spec
• Phase 2 design spec
• Phase 2 implementation plan