Skip to content

jasegehring/runwise

BranchesTags

Repository files navigation

Runwise

PyPI version CI License: MIT Python 3.10+

Token-efficient ML training run analysis for AI agents.

Runwise parses W&B and local training logs, generating condensed summaries optimized for LLM context windows. Designed for collaborative debugging between humans and AI assistants.

Features

  • Token-efficient output: Summaries designed to fit in LLM context without wasting tokens
  • Sparkline visualizations: Unicode trend graphs in ~10 tokens (e.g., loss: ▇▆▅▃▂▁ ↓)
  • Anomaly detection: Automatically flags spikes, overfitting, plateaus, gradient issues
  • W&B integration: Parse local wandb directories or use cloud API
  • W&B API support: Access remote runs without local sync (requires wandb package)
  • TensorBoard support: Optional parsing of tfevents files (requires tensorboard package)
  • Local log support: JSONL and other common log formats
  • Markdown export: --format md flag for GitHub issues, Notion, and documentation
  • Configurable schemas: Define metrics relevant to your project
  • MCP server: Direct integration with Claude Code and other MCP-compatible assistants
  • CLI tool: Quick analysis from the command line

Installation

pip install runwise

# Or from source:
git clone https://github.com/jasegehring/runwise
cd runwise
pip install -e .

Quick Start

CLI Usage

# List recent runs with sparkline trends
runwise list
runwise list --no-spark                        # Without sparklines (faster)

# Analyze latest run (includes anomaly detection + sparklines)
runwise latest
runwise latest -k loss,val_loss,accuracy       # Show only specific metrics
runwise latest --no-anomalies                  # Skip anomaly detection
runwise latest --no-spark                      # Skip sparklines

# Analyze specific run
runwise run abc123xyz
runwise run abc123xyz --no-spark --no-anomalies

# Compare two runs
runwise compare run_a run_b
runwise compare run_a run_b -f val             # Filter to validation metrics only
runwise compare run_a run_b -g                 # Group by metric prefix (train/, val/)
runwise compare run_a run_b -t 5               # Only show >5% changes
runwise compare run_a run_b -d                 # Show config differences

# Step-matched comparison (essential for curriculum learning)
runwise compare run_a@50000 run_b@50000        # Compare at same training step
runwise compare run_a@10000 run_b@20000        # Compare at different steps
runwise compare run_a@50000 run_b -f val       # Step-matched, validation only

# Show hyperparameters/config
runwise config                                 # Latest run
runwise config abc123                          # Specific run

# Show run context (name, notes, tags)
runwise notes                                  # Latest run
runwise notes abc123                           # Specific run

# Find best run by metric
runwise best val_loss                          # Lower is better (default)
runwise best accuracy --max                    # Higher is better
runwise best val_loss -n 20                    # Consider last 20 runs

# Get downsampled history (handles million-step runs efficiently)
runwise history                               # Auto-detects common metrics
runwise history -k loss,val_loss              # Specify keys explicitly
runwise history abc123 -k loss -n 100         # Specific run, 100 samples

# Get history statistics (even more compact)
runwise stats                                 # Auto-detects common metrics
runwise stats -k loss,val_loss,grad_norm      # Specify keys explicitly

# Stability analysis (rolling standard deviation)
runwise stability                             # Analyze training stability
runwise stability -k loss,val_loss            # Specify metrics
runwise stability -w 50                       # Custom window size (default: 100)
runwise stability --csv                       # Output as CSV

# List available metric keys
runwise keys                                   # Latest run
runwise keys abc123                            # Specific run

# Live training status (includes validation metrics)
runwise live

# Discovery and maintenance
runwise find "pattern"                         # Search by name, tag, group, notes
runwise sync                                   # List runs missing history files
runwise sync abc123                            # Sync specific run to recover history
runwise sync --all                             # Sync all unsynced runs
runwise watch                                  # Watch live output (tail -f with colors)
runwise watch -i 5                             # Update every 5 seconds

# Local JSONL logs (standalone files, not W&B)
runwise local                                  # List logs in logs/ directory
runwise local metrics.jsonl --keys             # List available metric keys
runwise local metrics.jsonl --history -k loss,val_loss  # Get history CSV
runwise local metrics.jsonl --stats -k loss,val_loss    # Get statistics
runwise local metrics.jsonl --stability -k loss         # Stability analysis

# TensorBoard support (requires: pip install tensorboard)
runwise tb                                     # List TB runs
runwise tb -r train_1                          # Summarize specific TB run

# W&B API support (requires: pip install wandb)
runwise api -p my-project                      # List runs (auto-detects project from wandb/)
runwise api -p my-project -r abc123            # Summarize specific run
runwise api -p my-project -r abc123 -k loss,val_loss  # Show specific metrics
runwise api -p my-project --best val/accuracy --max   # Find best run by metric
runwise api -p my-project -r run_id --history -k loss # Get metric history
runwise api -p my-project -c run1,run2         # Compare two runs
runwise api -p my-project -c run1,run2 -f val -t 5    # Compare with filtering
runwise api -p my-project --state running      # Filter by state

# Markdown output (for GitHub issues, Notion, etc.)
runwise latest --format md                     # Markdown summary
runwise list --format md                       # Markdown table
runwise compare run_a run_b --format md        # Markdown comparison

Python API

from runwise import RunAnalyzer, RunwiseConfig

# Auto-detect configuration from project
analyzer = RunAnalyzer()

# List runs
runs = analyzer.list_runs(limit=10)
print(analyzer.format_run_list(runs))

# Analyze latest
run = analyzer.get_latest_run()
print(analyzer.summarize_run(run))

# Compare runs
run_a = analyzer.find_run("abc123")
run_b = analyzer.find_run("def456")
print(analyzer.compare_runs(run_a, run_b))

# Get hyperparameters/config
print(analyzer.get_config(run))

# Find best run by metric
best, ranked = analyzer.find_best_run("val_loss", limit=10, higher_is_better=False)
print(analyzer.format_best_run("val_loss"))

# Get downsampled history (efficient for large runs)
history_csv = analyzer.get_history(run, ["loss", "val_loss"], samples=500)
print(history_csv)

# Get statistics only (most token-efficient)
stats = analyzer.get_history_stats(run, ["loss", "val_loss"])
print(stats)

# List available keys when you don't know what's logged
print(analyzer.list_available_keys(run))

Claude Code Skill (Recommended)

The most token-efficient way to use Runwise with Claude Code. Copy the skill to your project:

# From the runwise repo
cp -r .claude/skills/runwise /path/to/your/project/.claude/skills/

# Or create manually
mkdir -p .claude/skills/runwise
# Copy SKILL.md and examples.md from runwise repo

Claude will automatically discover and use the skill when you ask about training. The skill invokes CLI commands, which is more token-efficient than MCP (~500 tokens vs ~2000+).

Example prompts:

  • "How is training going?"
  • "Compare my last two runs"
  • "Is my model overfitting?"
  • "Show me the loss curve"

MCP Server (Alternative)

For programmatic access or non-Claude Code integrations, use the MCP server.

Add to your Claude Code MCP settings (~/.claude/settings.json):

{
    "mcpServers": {
        "runwise": {
            "command": "python",
            "args": ["-m", "runwise.mcp_server"],
            "env": {
                "RUNWISE_PROJECT_ROOT": "/path/to/your/project"
            }
        }
    }
}

Then in Claude Code, you can ask:

  • "Show me the latest training run"
  • "Compare runs abc and def"
  • "What's the live training status?"

When to use MCP vs Skill:

Use Case Best Option
Claude Code users Skill (lower tokens)
Other MCP-compatible tools MCP server
Programmatic/API access MCP server

Configuration

Create a runwise.json in your project root:

{
    "project_name": "My ML Project",
    "wandb_dir": "wandb",
    "logs_dir": "logs",
    "downsample_interval": 1000,
    "schema_inline": {
        "loss_key": "train/loss",
        "step_key": "_step",
        "primary_metric": "train/accuracy",
        "primary_metric_name": "Accuracy",
        "validation_sets": {
            "val": "Validation",
            "test": "Test"
        },
        "groups": [
            {
                "name": "training",
                "display_name": "TRAINING",
                "metrics": {
                    "train/loss": {"display": "Loss", "format": ".4f", "higher_is_better": false},
                    "train/accuracy": {"display": "Accuracy", "format": ".1%", "higher_is_better": true}
                }
            }
        ]
    },
    "anomaly_detection": {
        "spike_threshold": 3.5,
        "overfit_ratio_threshold": 1.5,
        "plateau_improvement_threshold": 0.01
    }
}

Anomaly Detection Thresholds

Customize anomaly sensitivity for your domain:

Setting Default Description
spike_threshold 3.5 MAD score threshold for spike detection (lower = more sensitive)
spike_window 100 Rolling window size for spike detection
overfit_ratio_threshold 1.5 Val/train ratio increase to flag overfitting (1.5 = 50% increase)
overfit_baseline_steps [100, 500] Step range for calculating baseline ratio
plateau_min_steps 500 Minimum steps before checking for plateau
plateau_improvement_threshold 0.01 Required improvement (0.01 = 1%)
gradient_vanish_threshold 1e-7 Gradient norm below this = vanishing
gradient_explode_multiplier 10.0 Gradient above 10x mean = exploding
throughput_drop_threshold 0.4 Throughput drop of 40% = system issue

Example: RL training (expect more variance)

{
    "anomaly_detection": {
        "spike_threshold": 5.0,
        "plateau_improvement_threshold": 0.005
    }
}

Example: Fine-tuning (strict monitoring)

{
    "anomaly_detection": {
        "spike_threshold": 2.0,
        "overfit_ratio_threshold": 1.2
    }
}

Or initialize with defaults:

runwise init --name "My Project"

W&B Best Practices

To ensure Runwise can read your training history, follow these W&B best practices:

Always Call wandb.finish()

The wandb-history.jsonl file (which stores training metrics) may not be fully written if your run is killed or crashes. Always ensure proper cleanup:

# Option 1: Context manager (recommended)
with wandb.init(project="my-project") as run:
    for step in range(1000):
        wandb.log({"loss": loss, "accuracy": acc})
    # wandb.finish() called automatically

# Option 2: Explicit finish
run = wandb.init(project="my-project")
try:
    for step in range(1000):
        wandb.log({"loss": loss, "accuracy": acc})
finally:
    wandb.finish()  # Always called, even on error

Sync Killed Runs

If a run was killed (Ctrl+C, OOM, etc.) before wandb.finish() was called:

# Sync a specific run directory
wandb sync wandb/run-20251214_120000-abc123xyz

# Sync all unsynced runs
wandb sync --sync-all

Troubleshooting Missing History

If runwise history shows "No history data found":

  1. Check the run directory: Look for wandb-history.jsonl in wandb/run-*/files/
  2. Use alternatives: runwise stats uses wandb-summary.json (final values only)
  3. Check run state: Killed/crashed runs may have incomplete data

Example Output

Run List (with sparkline trends)

RECENT RUNS (My ML Project):

ID           State     Date            Steps     Accuracy        Trend
------------------------------------------------------------------------
xyz789       RUNNING   2025-12-14      5,000       82.3%    ▇▆▅▄▃▂▁▁↓
abc123       FINISHED  2025-12-14     50,000       95.2%    ▇▅▃▂▁▁▁▁↓
def456       CRASHED   2025-12-13      2,341       45.0%    ▁▁▂▅▇▇▇▇↑

Run Summary (with anomaly detection)

=== My ML Project Run Summary ===
Run: abc123xyz | Step: 50,000 | Runtime: 12.5h

ANOMALIES:
  ! Overfitting: val/train ratio +35% vs baseline

TRAINING:
  Loss: 0.2341  ▇▆▅▄▃▂▂▁▁▁
  Accuracy: 87.3%  ▁▂▃▄▅▆▇▇▇█

VALIDATION:
  Validation: 85.2%
  Test: 83.7%

Config/Hyperparameters

CONFIG: abc123

  batch_size: 64
  dropout: 0.15
  learning_rate: 5.00e-04
  model: transformer
  num_layers: 8

Best Run

BEST RUN BY val_loss (from last 10 runs):

  BEST: abc123 = 0.1234

Rank   Run ID       State            val_loss
---------------------------------------------
1      abc123       finished           0.1234 *
2      def456       finished           0.1567
3      ghi789       crashed            0.8901

Stability Analysis

STABILITY ANALYSIS: abc123
Window size: 100 steps | Total: 50,000 steps

Metric           Slope/1k  Trend    Avg Std  Final Std     Status
--------------------------------------------------------------------
loss             -0.0023      ↓     0.0234     0.0012     STABLE
val_loss         -0.0015      →     0.0456     0.0089   MODERATE
grad_norm        +0.0089      ↑     0.3210     0.4521      NOISY

Slope/1k: change per 1000 steps (robust Theil-Sen estimate)
Trend: ↓=stabilizing  →=steady  ↑=destabilizing (variance trend)

Why "Token-Efficient"?

When collaborating with AI assistants on ML debugging, you often need to share training logs. Raw W&B exports or verbose logs can consume thousands of tokens, leaving less context for actual analysis.

Runwise generates summaries that:

  • Prioritize actionable metrics: Loss, accuracy, per-step breakdowns
  • Detect anomalies: Plateaus, regressions, divergence
  • Use compact formatting: Tables, aligned columns, minimal whitespace
  • Skip redundant data: No repeated headers, timestamps, or metadata

Handling Large Runs

The history command implements intelligent downsampling:

# A 1,000,000 step run returns exactly 500 data points
runwise history -k loss,val_loss -n 500

How it works:

  1. First pass: Count total lines (fast, no parsing)
  2. Calculate evenly-spaced sample indices
  3. Second pass: Parse only the sampled lines
  4. Output as CSV (most token-dense format)

This means a 10GB log file with millions of steps produces ~3000 tokens of output, regardless of size. The LLM never sees the raw file.

Contributing

Contributions welcome! Areas of interest:

  • Additional log format parsers (MLflow, etc.)
  • W&B API support for remote runs
  • GitHub Action for PR comments
  • Integration with other AI assistants

License

MIT

About

Token-efficient ML training run analysis for AI agents

Resources

Readme

License

MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases 5

v0.4.0: Step-matched Comparison & Usability Latest
Dec 15, 2025
+ 4 releases

Packages

 
 
 

Contributors

Languages