Tallyho

A practical benchmark visualizer for CSV snapshots with fast historical and trend analysis

Features

• normalized summary view across all benchmarks
• trend view across benchmark histories
• metric-level inspection for selected runs and overlays
• category filtering for grouped benchmark families
• project-specific labels, groups, and metric semantics via config
• persistent local view state (selected metrics, overlays, ranges)
• URL-hosted dashboard with quick CSV export of the visible series

Usage

Install and run as a command

Tallyho is designed for use as a global command.

Install from npm:

pnpm add -g @svjsonx/tallyho

For local development, use:

pnpm install
pnpm build
pnpm link -g .

Run it in a directory that contains benchmark data (for example ./benchmarks/):

tallyho ./benchmarks/

You can pass the port explicitly if needed:

tallyho ./benchmarks/ 3480

If a tallyho.json (or .tallyho.json) is present, its server.port value is used when no port argument is provided.

How Tallyho binds history

Each benchmark history is a directory containing snapshot folders. Snapshot folders are discovered from the target data path or from a configured data.snapshotRoot.

Example layout:

benchmarks/
  001_init_baseline_1f2a3b4/
    parse.csv
    layout/measure.csv
  002_next_candidate/
    parse.csv
    layout/measure.csv

Tallyho binds this as:

• snapshot index and description come from folder names
• all CSV rows inside each snapshot become runs
• run rows are grouped by benchmark + optional category
• each metric is averaged across runs to build a time-series entry
• histories are rendered as one series per benchmark/metric pair

Expected CSV shape

• one or more .csv files per snapshot
• optional category folders such as layout/ (detected from configured layouts)
• either:
  • explicit headers, or
  • legacy positional columns
• numeric columns become metrics unless listed in ignored columns

The parser is intentionally permissive: known metric columns get stable default labels, and unknown numeric columns are automatically included and grouped heuristically.

Configuration

Place tallyho.json (or .tallyho.json) in the path you run tallyho against.

• .tallyho.json has priority over tallyho.json.
• data.snapshotRoot controls where snapshot folders live, relative to the config directory (useful when there is a single wrapper folder).
• data.layouts controls which CSV files belong to each snapshot and whether category is derived from path segments.
• data.columns controls timestamp/benchmark/category aliases and ignored columns.
• metrics.columns overrides display label/group/kind metadata for specific metrics.
• metrics.primary sets preferred primary-metric order per benchmark.
• ui sets initial defaults for client state.
• server.port sets the default port when no port argument is provided.

If no config is present, Tallyho falls back to inference for both discovery and metric naming.

Example config file layout

{
  "version": 1,
  "project": {
    "name": "Generic benchmark history"
  },
  "data": {
    "snapshotRoot": ".",
    "snapshotPattern": "^(?<index>\\d+)[_-](?<description>.+?)(?:_(?<hash>[a-f0-9]{6,40}))?$",
    "layouts": [
      { "glob": "{snapshot}/{category}/*.csv", "categoryFromPath": true },
      { "glob": "{snapshot}/*.csv" }
    ],
    "columns": {
      "timestamp": ["timestamp", "ts", "time"],
      "benchmark": ["benchmark", "bench", "name", "test", "case"],
      "category": ["category", "suite", "group"],
      "ignore": ["goalpost", "run"]
    }
  },
  "server": {
    "port": 3478
  },
  "metrics": {
    "primary": ["exec_time_ms", "mean_time_ms", "total_time_ms"],
    "groups": [
      { "id": "time", "label": "Time", "order": 10 },
      { "id": "rendering", "label": "Rendering", "order": 20 },
      { "id": "other", "label": "Other", "order": 30 }
    ],
    "columns": {
      "exec_time_ms": { "label": "Exec time", "group": "time", "kind": "time", "unit": "ms" }
    }
  }
}

Version History

See release notes.

License

© 2026 Sven Johansson. ISC Licensed.