Skip to content
BranchesTags

Repository files navigation

Lab Notebook - Reproducible Experiment Runner yay!

A local-first CLI tool for running, tracking, and comparing computational experiments with full reproducibility guarantees.

Lab Notebook is designed for researchers and engineers who run computational experiments (ML training, simulations, bioinformatics pipelines) and need to track parameters, metrics, and artifacts without the complexity of cloud-based experiment tracking platforms.

Technical Stack

  • CLI Framework: Typer with Rich for beautiful terminal output
  • Configuration: YAML with Pydantic v2 validation
  • Storage: File-based JSON (no database required)
  • Web Dashboard: Pure HTML5/CSS3/JavaScript (zero dependencies, 100KB total)
  • Server: Python's built-in http.server for multi-device access
  • Scientific Computing: NumPy, Matplotlib for experiment implementations
  • Python: 3.10+ required

Design Decisions

Why File-Based Storage?

We chose a file-based approach over databases for several reasons:

  1. Portability: Your entire experiment history is just a folder you can zip, copy, or git-track
  2. No Infrastructure: No PostgreSQL, MongoDB, or MLflow server to maintain
  3. Transparency: Every run is a readable JSON file you can inspect with any text editor
  4. Simplicity: No migration scripts, no connection strings, no credentials
  5. Version Control: Git-friendly plain text format

This matches how researchers actually work - experiments are files, results are files, everything should be files.

Why Pure HTML/CSS/JS for Dashboard?

The web dashboard is intentionally framework-free:

  1. Zero Build Step: No npm, webpack, or bundlers - just open the HTML file
  2. Deployment Anywhere: Works on GitHub Pages, Netlify, any static host, or just double-click locally
  3. Tiny Size: Entire dashboard is ~100KB (vs. React apps that are 500KB+ minified)
  4. No Breakage: No dependencies means no security updates or breaking changes
  5. Universal Compatibility: Works in any browser from 2015 onward

Why CLI-First?

The CLI is the primary interface because:

  1. Scriptability: Integrates with existing shell scripts, cron jobs, and automation
  2. SSH-Friendly: Works over remote connections where GUIs don't
  3. Fast: lab list is faster than opening a web browser
  4. Composability: Pipe into grep, awk, or other Unix tools

The web dashboard is provided as a secondary interface for sharing and exploration.

Quick Start

Installation

# Clone the repository
git clone https://github.com/yourusername/ExperimentRunner.git
cd ExperimentRunner

# Install in development mode
pip install -e .

# Verify installation
lab --help

Run Your First Experiment

# Run a simple example
lab run examples/coin_flip.yaml

# View the results
lab last

# List all runs
lab list

Access Web Dashboard

# Generate and serve dashboard
lab serve

# Opens at:
#   Local: http://localhost:8000
#   Network: http://192.168.1.X:8000 (accessible from phone/tablet)

CLI Commands

Command Purpose Example
lab run Execute experiment from config lab run config.yaml
lab list Browse all runs with filters lab list -e ml_training --limit 10
lab show View run details lab show 1 (number-based navigation)
lab compare Compare two runs lab compare 1 2
lab last Quick check most recent run lab last
lab best Auto-find top performing runs lab best final_val_acc --top 5
lab table Compare multiple runs in table lab table final_val_acc -e ml_training
lab web Generate static HTML dashboard lab web -o docs/index.html
lab serve Start HTTP server for multi-device access lab serve -p 8000

See QUICKSTART.md for detailed command documentation.

Project Structure

ExperimentRunner/
├── labnotebook/              # Core package
│   ├── cli.py               # All 9 CLI commands (Typer app)
│   ├── config.py            # YAML config loading & Pydantic validation
│   ├── runner.py            # Run orchestration & metadata capture
│   ├── store.py             # File-based storage interface
│   ├── web.py               # Static HTML dashboard generator
│   └── server.py            # HTTP server for multi-device access
├── experiments/             # Example experiment implementations
│   ├── coin_flip.py         # Tutorial: Simple RNG
│   ├── ml_training.py       # ML hyperparameter tuning
│   ├── wright_fisher.py     # Population genetics simulation
│   └── sorting_benchmark.py # Algorithm comparison
├── examples/                # YAML config files
│   ├── coin_flip.yaml
│   ├── ml_lr_*.yaml
│   └── ...
├── runs/                    # Generated run outputs (git-ignored)
│   └── YYYY-MM-DD_HHMMSS_experiment_s42_hash/
│       ├── config.yaml      # Exact config used
│       ├── metadata.json    # Git commit, Python version, timestamps
│       ├── metrics.json     # Numerical results
│       ├── stdout.log       # Captured output
│       └── artifacts/       # Plots, CSVs, etc.
├── dashboard/               # Generated web dashboard
│   └── index.html           # Self-contained HTML file
├── docs/                    # Documentation
│   ├── QUICKSTART.md
│   ├── EXAMPLES.md
│   ├── ADVANCED_USAGE.md
│   ├── WEB_DEPLOYMENT.md
│   ├── VSCODE_INTEGRATION.md
│   ├── MULTI_DEVICE_ACCESS.md
│   └── WEB_AND_VSCODE_SUMMARY.md
└── pyproject.toml           # Package configuration

Documentation

Getting Started

  • QUICKSTART.md - Complete CLI command reference with examples
  • EXAMPLES.md - Step-by-step walkthroughs of example experiments

Advanced Features

IDE Integration

Use Cases

Machine Learning Hyperparameter Tuning

# Run experiments with different learning rates
lab run examples/ml_lr_small.yaml
lab run examples/ml_lr_medium.yaml
lab run examples/ml_lr_large.yaml

# Automatically find best validation accuracy
lab best final_val_acc --min -e ml_training --top 3

# Compare all runs in a table
lab table final_val_acc -e ml_training

Algorithm Benchmarking

# Run sorting algorithm comparison
lab run examples/sorting_benchmark.yaml

# View results
lab last

# Compare with previous benchmark
lab compare 1 2

Simulation Studies

# Run population genetics simulations with different parameters
lab run examples/wright_fisher_neutral.yaml
lab run examples/wright_fisher_selection.yaml

# Generate dashboard for lab meeting
lab serve

# Access from any device on network at http://192.168.1.X:8000

Reproducibility Verification

Every run captures:

  • Exact config (YAML snapshot)
  • Git commit hash (if in repo)
  • Python version
  • Random seed
  • Full stdout/stderr logs
  • Timestamps (start, end, duration)

This means: same config + same seed = identical results

# Run experiment
lab run config.yaml

# Later, reproduce exact results
lab show 1  # Check git commit
git checkout <commit>
lab run config.yaml  # Will produce identical output

Local Development

Prerequisites

  • Python 3.10 or higher
  • pip or uv for package management
  • Git (optional, for metadata capture)

Setup

# Clone repository
git clone https://github.com/yourusername/ExperimentRunner.git
cd ExperimentRunner

# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install in editable mode with all dependencies
pip install -e .

# Run tests (if implemented)
pytest

VSCode Integration

Lab Notebook integrates seamlessly with VSCode:

# Copy VSCode configuration files
cp -r .vscode-example .vscode

# Reload VSCode
# Now you can:
# - Ctrl+Shift+P -> Tasks: Run Task -> "Lab: List Runs"
# - F5 to debug experiments
# - Ctrl+Shift+L to list runs (custom shortcut)

See VSCODE_INTEGRATION.md for complete setup.

Web Dashboard Deployment

Local (Instant)

# Generate and open dashboard
lab web
python -m http.server -d dashboard 8000
# Open: http://localhost:8000

GitHub Pages (2 minutes)

# Generate in docs folder
lab web -o docs/index.html

# Push to GitHub
git add docs/
git commit -m "Add dashboard"
git push

# Enable in: Settings -> Pages -> Source: /docs folder
# Live at: https://yourusername.github.io/ExperimentRunner/

Netlify/Vercel (30 seconds)

# Generate dashboard
lab web -o dashboard/index.html

# Drag and drop 'dashboard' folder to Netlify/Vercel
# Instantly deployed!

See WEB_DEPLOYMENT.md for detailed deployment instructions.

Multi-Device Access

Access your experiments from any device on your network:

# On your laptop
lab serve

# Output shows:
#   This computer:    http://localhost:8000
#   Other devices:    http://192.168.1.100:8000

# Now open that IP on:
# - Your phone
# - Tablet
# - Other laptops
# - Any device on same WiFi

No installation needed on viewing devices - just a web browser!

See MULTI_DEVICE_ACCESS.md for complete guide.

Security & Privacy

What's Tracked

The dashboard shows:

  • Run IDs
  • Experiment names
  • Metrics (numerical values)
  • Timestamps
  • Seeds
  • Run status (success/failed)

What's NOT Exposed

The dashboard does NOT include:

  • Raw data files
  • Source code
  • Config file contents
  • Actual artifacts
  • File system paths
  • Environment variables

Network Security

  • lab serve binds to 0.0.0.0 (all interfaces) but is only accessible from your local network
  • NOT exposed to the internet unless you explicitly set up port forwarding
  • For public deployment, use GitHub Pages with private repos or add authentication

Adding Password Protection

See WEB_DEPLOYMENT.md for client-side and server-side authentication options.

Customization

Custom Experiments

Create your own experiment by following this pattern:

# experiments/my_experiment.py
import random
from labnotebook import RunContext

def run(ctx: RunContext):
    """Your experiment logic."""
    # Access config
    param = ctx.config.params.get("my_param", 1.0)

    # Set seed for reproducibility
    random.seed(ctx.config.seed)

    # Run your experiment
    result = your_computation(param)

    # Log metrics
    ctx.log_metric("accuracy", result)

    # Save artifacts
    ctx.save_artifact("plot.png", your_plot())

Then create a YAML config:

# examples/my_experiment.yaml
experiment: my_experiment
seed: 42
params:
  my_param: 1.5
tags:
  - custom
note: "Testing my custom experiment"

Run it:

lab run examples/my_experiment.yaml

Dashboard Customization

Edit labnotebook/web.py to customize:

  • Color schemes (stat card gradients)
  • Table columns
  • Logo/branding
  • Additional filters

The entire dashboard is defined in HTML_TEMPLATE string - no build step required!

Comparison with Other Tools

Feature Lab Notebook MLflow Weights & Biases Sacred
Setup pip install -e . Requires server Cloud account pip install sacred
Storage Local files Database required Cloud only MongoDB required
Web UI Static HTML React app Cloud dashboard No built-in UI
CLI Full-featured Limited No CLI Python API only
Offline 100% offline Requires server Internet required Needs MongoDB
Deployment GitHub Pages Self-host server Managed cloud N/A
Size ~100KB dashboard ~50MB server N/A ~5MB

Lab Notebook trades advanced features (hyperparameter search, model registry, collaboration tools) for simplicity and zero infrastructure.

Contributing

Contributions welcome! Areas for improvement:

  • Additional example experiments (NLP, computer vision, optimization)
  • Export to other formats (CSV, Parquet, HDF5)
  • Parameter sweep automation
  • Artifact diff visualization
  • Custom dashboard themes
  • Integration tests

License

MIT License - see LICENSE file for details.

Acknowledgments

Built with:

Inspired by MLflow, Sacred, and Weights & Biases - but designed for researchers who prefer files over databases.

Support

  • Documentation: See docs/ folder
  • Issues: GitHub Issues
  • Questions: GitHub Discussions

Lab Notebook: Simple, local, reproducible experiment tracking.

About

Experiment Runner

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages