Skip to content

TUI Dashboard

Jump to bottom
Dave Graham edited this page Mar 18, 2026 · 3 revisions

TUI Dashboard

A real-time terminal dashboard for monitoring autoresearch training runs on Apple Silicon, built with Textual.

Available on: master branch

Screenshot

TUI Dashboard

Dashboard showing a training run in progress on Apple M5 Max — live metrics updating every ~1 second.

Features

Panel Description
Training Live progress bar, training loss, learning rate, tok/sec throughput, MFU (Model FLOPS Utilization), ETA countdown
Hardware Apple Silicon chip name, unified memory total and usage bar, GPU core count, estimated peak bf16 TFLOPS
Experiments History table loaded from results.tsv — shows experiment ID, status (keep/discard/baseline), val_bpb, memory, throughput, steps
Activity Log Timestamped scrollable log of startup info, model configuration, and final evaluation results

Installation

# From the repo root
uv sync --extra tui              # Textual only
# or
uv sync --extra all              # All backends + TUI + agent

Usage

# Launch dashboard with MLX training (default)
uv run dashboard.py

# Launch with MPS (PyTorch) training
uv run dashboard.py train.py

# Watch mode — show dashboard without starting training
uv run dashboard.py --watch

# Training scripts still work standalone (no TUI changes required)
uv run train_mlx.py

Keybindings

Key Action
q Quit (kills training subprocess)
d Toggle dark/light mode
r Reload experiments table from results.tsv

Architecture

The dashboard runs the training script as a subprocess and parses its stdout in real-time. This means:

  • Zero changes to training code — train_mlx.py and train.py are unmodified
  • Process isolation — a training crash doesn't kill the dashboard
  • Both backends supported — works with MLX and MPS identically

The training scripts output step metrics using print(..., end="", flush=True) with \r carriage returns for in-place line updates. The TUI uses a threaded byte-by-byte reader to capture each flush immediately, splitting on \r and \n to extract individual step updates.

File Structure

dashboard.py             Entry point
tui/
  __init__.py            Package init
  app.py                 Textual Application — layout, subprocess management, keybindings
  widgets.py             TrainingPanel, HardwarePanel, ExperimentsTable, ActivityLog
  parser.py              Regex parser for training stdout (step metrics + final summary)
  hardware.py            Apple Silicon hardware detection (wraps backends/__init__.py)
  experiments.py         results.tsv loader for experiment history table
  styles.tcss            Textual CSS for panel layout and styling

Technical Details

Output parsing: The parser uses regex to extract 9 per-step metrics from lines like:

step 00192 (62.3%) | loss: 4.168331 | lrm: 0.66 | dt: 1001ms | tok/sec: 32,737 | mfu: 23.0% | epoch: 1 | remaining: 118s

Final summary parsing: After training completes, the script outputs a --- separator followed by key-value pairs (val_bpb, peak_vram_mb, mfu_percent, etc.) which are parsed and displayed in the Training panel's completion view.

Hardware detection: Uses sysctl to identify the Apple Silicon chip, GPU core count, and total memory. Estimates peak bf16 TFLOPS based on per-generation FLOPS-per-core values (M1: 0.5T, M2: 0.55T, M3: 0.65T, M4: 0.7T, M5: 0.85T).

Apple Silicon limitations: GPU temperature and load percentage are not available on macOS without sudo powermetrics. The dashboard focuses on memory usage (the critical metric for unified memory pressure) and omits temp/load.

Troubleshooting

Dashboard starts but training never progresses

The most common cause is memory pressure. If the auto-detected hyperparameters are too large for your machine, the first mx.eval() call will thrash swap and appear to hang forever. Check Activity Monitor — if the process shows memory usage significantly above your physical RAM, the batch size is too large.

Fix: edit the hyperparameter defaults in backends/__init__.py → suggest_hyperparameters(), or override directly in train_mlx.py. The characterization sessions found these optimal values:

Chip Memory Batch Size Device Batch Depth
M5 Max 64 GB 32,768 16 8
M4 Pro 24 GB 8,192 8 6
M1 Max 64 GB 16,384 8 8

Step metrics appear delayed by one step

This is expected. The training script uses \r (carriage return) with no trailing newline. The byte-by-byte reader accumulates characters until the next \r arrives, so there's always a one-step display lag. At ~1 step/second this is imperceptible.

Dependencies

  • Textual >= 3.0.0 (includes Rich as a transitive dependency)
  • Pure Python, no binary dependencies
  • Added as optional tui extra in pyproject.toml

Clone this wiki locally