Skip to content

v0.5.1

Latest

Choose a tag to compare

@github-actions github-actions released this 09 Nov 10:13
· 4 commits to main since this release
d6c5e98

TraceML 0.5.1 adds aggregator-side live telemetry export.

Live telemetry export

Export per-step, per-rank timings to JSONL while training:

traceml run --exporter jsonl train.py

By default, records are written to:

logs/<run-name>/aggregator/export/step_time.jsonl

TraceML also supports custom exporters for sending records directly to an experiment-management or observability platform:

traceml run --exporter your_package.your_module:make_exporter train.py

Export runs through the central aggregator in background batches. Available streams include step timing, step memory, process metrics, and system metrics. Export failures do not fail the training run.

Other improvements

  • Final reports now read from one consistent SQLite snapshot.
  • Improved step-measurement and export draining during shutdown.
  • Corrected aggregation of CPU-only and GPU-timed calls.
  • Improved distributed rank identity and configuration validation.
  • Standardized memory units across reports.

Compatibility note

Raw telemetry now uses integer Unix timestamps in nanoseconds, and the final-summary schema is now 2.0. Start a fresh run after upgrading and use the same TraceML version for workers and the aggregator. Existing saved summaries remain readable.

Upgrade

pip install --upgrade "traceml-ai==0.5.1"

View the full changelog

Contributors

Thanks to:

  • @Pendu for implementing live telemetry export, reporting improvements, and release hardening.
  • @abhinavsriva for the RF-DETR example and release coordination.
  • @dependabot for dependency updates.

View the full changelog