Skip to content

beta-260617: Support per-job progress JSONL & improve metrics

Choose a tag to compare

@LingyeSoul LingyeSoul released this 17 Jun 08:50
Ensure command-style training jobs use a daemon-owned per-job progress.jsonl instead of the shared output/logs fallback, and wire that path end-to-end. Added detection in JobManager to inject --progress_jsonl for tasks.py training commands and tuned TQDM env vars. Prevent run() from unconditionally redirecting grandchild stdout to DEVNULL when our stdout is already a regular file (avoids dropping tqdm redraws under the detached daemon). Overhauled TaskService: derive per-job JSONL paths, accept explicit --progress_jsonl, map accelerator-style JSONL keys to the dashboard fields, emit metrics per JSONL step (removed the problematic debounce), and compute timing/speed/ETA from JSONL timestamps. Updated Task dataclass to track progress timing and added helpers for formatting. Included regression tests covering command progress wiring and stdout/devnull behavior, and updated existing metrics tests to reflect the new JSONL handling and timing logic.