Repository navigation
TraceML 0.5.1 adds aggregator-side live telemetry export.
Live telemetry export
Export per-step, per-rank timings to JSONL while training:
traceml run --exporter jsonl train.pyBy default, records are written to:
logs/<run-name>/aggregator/export/step_time.jsonl
TraceML also supports custom exporters for sending records directly to an experiment-management or observability platform:
traceml run --exporter your_package.your_module:make_exporter train.pyExport runs through the central aggregator in background batches. Available streams include step timing, step memory, process metrics, and system metrics. Export failures do not fail the training run.
Other improvements
- Final reports now read from one consistent SQLite snapshot.
- Improved step-measurement and export draining during shutdown.
- Corrected aggregation of CPU-only and GPU-timed calls.
- Improved distributed rank identity and configuration validation.
- Standardized memory units across reports.
Compatibility note
Raw telemetry now uses integer Unix timestamps in nanoseconds, and the final-summary schema is now 2.0. Start a fresh run after upgrading and use the same TraceML version for workers and the aggregator. Existing saved summaries remain readable.
Upgrade
pip install --upgrade "traceml-ai==0.5.1"Contributors
Thanks to:
- @Pendu for implementing live telemetry export, reporting improvements, and release hardening.
- @abhinavsriva for the RF-DETR example and release coordination.
- @dependabot for dependency updates.