Caliper - ML Experiment Tracker

Caliper is an offline-first machine learning experiment tracker and dashboard with built-in convergence forecasting and inline step-level annotations.

No account registration is required. Caliper functions 100% locally with zero cloud dependencies.

---

Architecture & How It Works

Caliper uses a local file-system synchronization architecture to provide high-performance logging with zero network latency.

+------------------+                   +--------------------+
|  Python Script   |                   |  Local Directory   |
| (import caliper) | --[Log Metrics]-->|   .caliper/runs/   |
+------------------+                   +--------------------+
                                                 |
                                         [CLI watches files]
                                                 v
+------------------+                   +--------------------+
|  Local Dashboard |                   |    Caliper CLI     |
|   (Browser UI)   | <---[WebSockets]--|    (Local Host)    |
+------------------+                   +--------------------+

1. Python Client Logger (caliper-py): The logger writes metrics incrementally to local JSONL files in a .caliper/ directory inside your workspace.
2. Local Server & CLI: A CLI utility watches the .caliper/ directory and serves the compiled dashboard on a local port.

PyPI Distribution

To integrate Caliper into Python training scripts, a PyPI distribution is used:

pip install caliper-py

This package installs the caliper logging client and the CLI utility caliper.

---

Walkthrough

1. Track Experiments in Python

Import caliper and track your training run metrics within your training loop:

import caliper

# Initialize a run under a specific project
run = caliper.init(
    project="CIFAR-100 Baselines",
    name="resnet50-cosine-lr",
    hyperparams={
        "model": "ResNet-50",
        "optimizer": "Adam",
        "learningRate": 0.001,
        "batchSize": 64,
        "epochs": 50,
        "scheduler": "cosine"
    }
)

for epoch in range(50):
    for step, (x, y) in enumerate(train_loader):
        loss, acc = train_step(x, y)
        
        # Log training step metrics (appended to local JSONL file)
        run.log({
            "step": global_step,
            "epoch": epoch,
            "trainLoss": loss.item(),
            "trainAcc": acc.item()
        })
        
    # Log validation metrics
    val_loss, val_acc = validate()
    run.log({
        "step": global_step,
        "valLoss": val_loss,
        "valAcc": val_acc
    })
    
    # Add step-level annotation notes (marginalia)
    if epoch == 15:
        run.annotate(
            step=global_step,
            text="Warmup complete - transitioning to cosine decay",
            metric="valLoss"
        )

2. View the Local Dashboard

Once metrics are logged to .caliper/, launch the interactive dashboard using the CLI:

caliper ui

This starts the local web server on http://localhost:5173/ and visualizes your runs.

---

Dashboard Technical Features

• Convergence Forecasting: Fits a 3-parameter exponential decay curve to the validation loss data to project convergence and detect anomalies (e.g., plateauing, divergence, oscillation).
• Inline Annotations (Marginalia): Visualizes step-level annotation notes directly on metric charts and maps them to a side-panel timeline.
• Project Filtering: Groups runs by project name with sidebar filtering capabilities.
• Multi-run Comparison: Compares up to 4 runs side-by-side with automatic parameter difference highlighting.

---

Developer Installation (React/TypeScript)

If you are developing or contributing to the React dashboard interface:

1. Clone the repository

git clone https://github.com/verz0/Caliper.git
cd Caliper

2. Install dashboard dependencies

npm install

3. Run dashboard in development mode

npm run dev

---

License

Licensed under the MIT License.