Litracer

Convert LitData enable_tracer() logs into Chrome / Perfetto traces.

Repo: Lightning-AI/litracer. LitData cookbook: Debug & Profile.

---

Install

From this repo (recommended while the Go module path is still github.com/deependujha/litracer):

git clone https://github.com/Lightning-AI/litracer.git
cd litracer
go build -o litracer .

Via Go (module path; pulls the published module, not necessarily this GitHub default branch):

go install github.com/deependujha/litracer@latest

Requires Go 1.23+ (go.mod language version). CI and the module toolchain use Go 1.26.6. go install github.com/Lightning-AI/litracer@latest will not work until go.mod is renamed to that module path.

---

Usage with LitData

Call enable_tracer() once per process, before creating the DataLoader. Delete an existing litdata_debug.log before re-tracing (the handler appends).

from litdata.debugger import enable_tracer

enable_tracer(level="chunk", log_file="litdata_debug.log")
# level="batch" | "chunk" (default) | "sample" | "debug" | "off"
# categories=["download", "read", "delete"]  # optional override

Level	What you get
batch	Epoch + per-batch spans, plus crashes
chunk	+ download, read, delete, decompress, prefetch
sample	+ per-__getitem__ spans (high volume)
debug	+ .cnt lock refcount spans
off	Disable

python train.py
litracer --quiet --validate -o litdata_trace.json litdata_debug.log
litracer --quiet --cat download,read,delete -o io.json litdata_debug.log
# open the json in https://ui.perfetto.dev

--quiet (or TERM=dumb) skips the progress UI and prints a one-line summary with per-category durations. --validate checks the JSON parses and event counts match.

Flags

Usage:
  litracer [flags] <log_file>

Flags:
  -h, --help            help
  -o, --output string   Path to the output trace file (default "litdata_trace.json")
  -q, --quiet           Skip the progress UI and print a one-line summary
      --validate        Parse the output JSON and exit non-zero if it is invalid
      --cat string      Keep only these categories (download,read,delete,decompress,batch,sample,epoch,lock,crash)
      --no-complete     Keep separate B/E events instead of compacting to complete (X) spans
  -w, --workers int     Parse goroutines (default: GOMAXPROCS)
  -s, --sink int        Deprecated: ignored

--cat and --no-complete apply in both quiet and TUI modes.

---

Log format

Each non-empty line is one Chrome event as semicolon-separated key: value pairs. LitData writes exactly one line per event (no traceback bodies). Timestamps are Chrome microseconds (created * 1e6); values older than 1e12 are treated as Unix seconds and multiplied.

ts:<microseconds>;PID:<pid>; TID:<tid>; name: download;ph: B;cat: download;cname: rail_load;chunk: 12;worker_rank: 1;...

Required keys: name, ph. Everything else becomes a Perfetto arg (or a well-known field: pid, tid, cat, cname, s, dur).

ph	Meaning
B / E	Begin / end of a duration span (same name + pid + tid)
X	Complete span (already compacted; dur set)
I	Instant (crashes). Scope defaults to t (thread)
M	Metadata (process_name / thread_name) written by Litracer

Lines without ph (Python Traceback …, blank lines, noise) are skipped.

Matched B/E pairs are compacted to complete (ph: X) spans unless --no-complete. That keeps Perfetto files smaller.

Stable event names (LitData current tracer)

Indexes live in args, not in the event name, so Perfetto groups all downloads together.

name	cat	When
download	download	Chunk GET to cache
prefetch	download	Async prefetch gather
read	read	Chunk mmap / decode
delete	delete	Evict a consumed chunk
decompress	decompress	zstd (or other) inflate
batch	batch	DataLoader batch
dataloader / combined	epoch	Epoch span
sample	sample	Per-item __getitem__
lock	lock	.cnt increment/decrement (level="debug")
crash	crash	Prepare-thread death (ph: I). Traceback is also printed to stderr, not the log file

Every event also carries dist_world_size, dist_global_rank, dist_num_nodes, worker_world_size, worker_rank.

Older logs

Still convert:

• Names like download_chunk_12, read_chunk_3, prepare_chunks_thread_crashed_TypeError
• Unix-second timestamps (ts:1750000000.5)
• Missing cat: (inferred from name)

---

Interpreting the summary

lines=698 events=360 skipped=0 B=0 E=0 X=349 I=1 unmatched_B=0 unmatched_E=0 crashes=1 processes=9 duration_ms=… download=…ms read=…ms delete=…ms

• skipped — lines with no ph (tracebacks, blanks).
• unmatched_B / unmatched_E — duration events with no pair. A leftover open read at worker shutdown used to cause this; current LitData closes the last read span.
• crashes — instant crash / *crashed* events. Also look at the training process stderr for [litdata] PrepareChunksThread CRASHED.

Open litdata_trace.json in ui.perfetto.dev (preferred) or chrome://tracing. For files > 2GB, see Perfetto large traces.

---

Develop

go test ./...
go build -o litracer .

CI (.github/workflows/test.yml) runs go test ./... and go build with Go 1.26.6 on pull requests and on pushes to main.