refactor(megatron_training_lib): add ordered fallback chains for log parsing [AIMVT-161]

[atnair-amd]

Summary

Closes [AIMVT-161]. Refactors the inline regex parsing/progress/NaN checks in cvs/lib/megatron_training_lib.py into ordered fallback chains and pure module-level helpers. Each chain is seeded with [new, old] so newer Megatron output (throughput per GPU (TFLOP/s/GPU): N) is preferred but older output (throughput per GPU: N) still parses correctly. No config or doc changes — pattern lists are intentionally library-internal.

Why

The parsing regex was edited in place when newer Megatron builds changed the output format (throughput per GPU: to throughput per GPU (TFLOP/s/GPU):), losing the ability to parse logs from the older builds. A fallback chain is the structural fix: future format changes append a new pattern instead of overwriting an old one, and old logs keep parsing.

The progress regex was also buried inside poll_for_training_completion's time.sleep(80)-then-loop machinery, which made it un-unit-testable without monkeypatching sleeps + phdl.exec + scan_for_training_errors + get_training_results_dict. Extracting pure helpers makes both the parsing and progress/NaN checks testable with a single string fixture.

What changed

cvs/lib/megatron_training_lib.py:

Three new module-level constants — ordered pattern chains, [new, old]:

TRAINING_RESULT_PATTERNS = {
    'throughput_per_gpu': [
        r'throughput per GPU(?:\s*\([^)]*\))?\s*:\s+([0-9\.]+)',
        r'throughput per GPU:\s+([0-9\.]+)',
    ],
    'tokens_per_gpu': [r'tokens\/GPU\/s:\s+([0-9]+)'],
    'mem_usage': [r'mem usages:\s+([0-9\.]+)'],
    'elapsed_time_per_iteration': [r'elapsed time per iteration \(ms\):\s+([0-9\.]+)'],
}

TRAINING_PROGRESS_PATTERNS = [
    r'throughput per GPU(?:\s*\([^)]*\))?\s*:|tokens\/GPU\/s\s+[0-9]+',
    r'throughput per GPU:|tokens\/GPU\/s\s+[0-9]+',
]

TRAINING_NAN_PATTERNS = [...]   # same shape; behavior-preserving

Three new pure helpers — first non-empty match wins:

def _parse_training_results(output: str) -> dict: ...
def _is_training_complete(output: str) -> bool: ...
def _has_nan_inf_results(output: str) -> bool: ...

• get_training_results_dict reduces to a thin wrapper: fetch the log via phdl.exec, then _parse_training_results(output).
• poll_for_training_completion replaces its inline re.search(...) calls with _is_training_complete(output) and _has_nan_inf_results(output).

cvs/lib/unittests/test_megatron_training_lib.py — append two test classes (4 tests total):

• TestTrainingLogParsing.test_parse_new_format — parser extracts every metric from new-format input.
• TestTrainingLogParsing.test_parse_old_format_falls_back — parser extracts every metric from old-format input.
• TestProgressDetection.test_handles_new_format and test_handles_old_format — exercise the separate TRAINING_PROGRESS_PATTERNS chain (different code path from parsing).

[anujmittal-amd]

Other than the comment below along with the hca pattern comment; LGTM for merge.