Skip to content

v0.3.0 - Round-Robin Benchmark Mode

Choose a tag to compare

View all tags
@JackWeiw JackWeiw released this 23 Jul 11:49
· 490 commits to main since this release
v0.3.0
5f1d959

🚀 v0.3.0 - Round-Robin Benchmark Mode

This release adds round-robin benchmark mode for E2B sandbox testing, enabling memory migration stress testing through controlled sandbox group rotation.

Key Features

🔄 Round-Robin Sandbox Rotation

  • Group-based execution: Splits sandboxes into equal groups (configurable via round_count or round_size)
  • Cycling support: Automatically cycles through groups when rounds exceed group count, enabling continuous testing within test_duration
  • Per-round metrics: Independent statistics tracking for each round with accurate latency extraction

🌊 Wave-Based Warmup for Large Scale Tests

  • Automatic batching: When total_count > 100, warmup executes in waves of 100 sandboxes each
  • Cross-process state: Supports --detect mode by splitting detected sandboxes into waves
  • Memory preheating: Opens multiple tabs per sandbox, executes operations to allocate memory before benchmark

📊 Tab Operation Benchmark Mode

Each benchmark round executes:

  • New tab → Opens a new browser tab with URL from browser_urls (round-robin selection)
  • Snapshot → Captures DOM state
  • Click → Interacts with first available element
  • Screenshot → Captures visual state

📈 Multi-Level Metrics Collection

Level Metrics Purpose
Per-Sandbox Task count, success rate, latencies, step timing Individual sandbox health
Per-Round Task delta, success delta, latency distribution Round comparison
Per-Step open_tab, page_load, snapshot, click, screenshot timing Performance breakdown
Aggregate Total tasks, overall success rate, P99 latency Summary statistics

🔍 Enhanced Error Diagnostics

  • Detailed error recording with exit code, stderr, stdout, and context
  • Automatic error type classification (tab operation errors, legacy browser errors, timeout, other)
  • Thread-safe metrics collection with threading.Lock()

💾 Swap Monitoring Enhancement

  • Swap cache and activity metrics in vm_monitor
  • Swap_Timeline sheet with chart in Excel summary
  • Improved swap display formatting for real-time and summary report

Configuration

test:
  benchmark_mode: "round_robin"   # Enable round-robin mode

  # Choose ONE:
  round_count: 5                  # Number of groups (auto-calculated size)
  # round_size: 20                # Sandboxes per group (auto-calculated count)

  round_interval: 30              # Gap between rounds (seconds)

Bug Fixes

  • Round comparison latency statistics correction
  • Round baseline timing for accurate task counts (sentinel entry)
  • Step timing data type fix for click/screenshot errors
  • Config priority enforcement: CLI > YAML > defaults
  • Wave mode duplicate sandbox creation fix
  • Division by zero when tab_ids is empty
  • Warmup delay config override fix

Testing

  • 162 tests total (39 new for round-robin features)
  • All tests pass ✅

Full Changelog

v0.2.0...v0.3.0

Assets 2
Loading