Skip to content

🚧 RLM Phase 1.1 Alpha - DO NOT USE IN PRODUCTION

Pre-release
Pre-release

Choose a tag to compare

View all tags
@vmlinuzx vmlinuzx released this 26 Jan 03:55
· 24 commits to feat/rlm-config-nested-phase-1x since this release
v2.0.0-alpha.1
6de0b55

⚠️ ALPHA RELEASE - KNOWN SECURITY VULNERABILITIES ⚠️

This is an experimental release for testing and feedback. DO NOT use in production.

🎯 What Works

Proven Performance (Real-World Testing)

  • 83.4% token reduction vs baseline (agent with filesystem tools, no caching)
  • 85% cache hit rate across multiple runs
  • Identical quality - found same security bugs as baseline

Note: Traditional RAG (loading full codebase upfront) would be significantly MORE expensive than our baseline, making RLM's advantage even larger.

Benchmark Results (Actual Test Data)

Test Setup:

  • Task: Comprehensive security audit + static analysis + behavioral testing
  • Model: gemini-3-flash-preview (Google)
  • Repo: LLMC codebase (~1000 files, 10k+ functions)
  • Methodology: Adversarial testing demon (Rem) - same task, different modes

Raw Numbers:

Metric Baseline (No RLM) RLM Run 1 RLM Run 2 RLM Average
Total Tokens 2,034,594 2,521,312 2,007,473 2,264,393
Cache Reads 0 2,114,715 1,711,560 1,913,138
Cache Hit Rate 0% 84.5% 85.3% 84.9%
Effective Input 2,018,599 388,768 279,631 334,200
Output Tokens 7,166 4,584 8,412 6,498
Duration 228s (3.8min) 372s 390s 381s (6.4min)
Tool Calls 35 34 38 36
Cost (standard) ~$0.20 ~$0.03 ~$0.02 ~$0.025

Key Finding: RLM reduces effective tokens from 2,018,599 → 334,200 (83.4% reduction)

What the baseline is:

  • Gemini with file system access (read_file, list_directory tools)
  • NO RLM lazy loading
  • NO caching
  • The agent reads files on-demand but pays full cost every time

Why this matters:

  • Traditional RAG loads entire codebase into context upfront (even more expensive)
  • Our baseline is already "smart" (selective file reading)
  • RLM still achieves 83% reduction through intelligent caching

Quality (Bug Findings - Both Modes Identical)

Both baseline and RLM found the exact same CRITICAL security vulnerabilities:

  • Path traversal (arbitrary file read)
  • Sandbox escape (RCE via os.system)
  • Missing validation in config classes
  • Token budget bypass

Quality: Identical. Cost: 83% lower. Cache hit rate: 85%.

❌ Known Critical Issues

These are DOCUMENTED vulnerabilities. Do not use this in production:

  1. Path Traversal - Arbitrary file read (CRITICAL)
  2. Sandbox Escape - RCE via os.system (CRITICAL)
  3. Missing Validation - McpConfig lacks Pydantic validation
  4. Budget Bypass - Token estimation vulnerable to unicode attacks

All issues found and documented by adversarial testing demon.
See: tests/REPORTS/current/rem_testing_2026-01-25.md

📦 Installation

pip install git+https://github.com/vmlinuzx/llmc@v2.0.0-alpha.1

📚 Documentation

🔬 What's New

RLM MCP Tool

  • Lazy loading with intelligent caching (85% hit rate)
  • MCP integration for Gemini/Claude
  • Budget tracking and governance
  • Navigation tools (treesitter-based)

Benchmark Framework

  • Bedilia - A/B testing orchestrator (baseline vs RLM)
  • Adversarial demons - Automated bug hunters
  • Token tracking with stream-json output
  • Automated comparison reports

How RLM Works

Instead of loading entire codebase into context:

  1. Load minimal initial context
  2. LLM requests specific files/functions via MCP tools
  3. Context cached aggressively (85% reuse across turns)
  4. Pay only for net-new information (15% of tokens)

Think "LSP for LLMs" - lazy loading + intelligent caching.

Comprehensive Testing

  • 43/43 happy path tests passing
  • Security PoC tests documenting known vulnerabilities
  • Integration tests with DeepSeek
  • Real-world benchmark validation vs baseline

🎯 For Early Adopters

If you want to test RLM despite the known issues:

# Clone the feature branch
git clone https://github.com/vmlinuzx/llmc.git
cd llmc
git checkout feat/rlm-config-nested-phase-1x

# Install
pip install -e .

# Run benchmarks yourself
./scripts/validate_rlm_system.sh

🚨 SECURITY WARNING

This release contains documented security vulnerabilities. It is released as alpha to:

  • Demonstrate the core concept works (83% token reduction, 85% cache hit rate)
  • Get early feedback on the approach
  • Allow community testing and contributions
  • Share benchmark framework for others to validate

Use at your own risk. Do not deploy to production.

💬 Feedback

Issues, questions, or want to help fix the security bugs? Open an issue or PR!

Links:

  • Feature branch: feat/rlm-config-nested-phase-1x
  • Research paper: DOCS/research/Recursive Intelligence in Repository scale environments.txt
  • Benchmark results: DOCS/research/BENCHMARK_STATUS.md
  • Demon findings: tests/REPORTS/current/rem_testing_2026-01-25.md
Assets 2
Loading