feat(ci): add multi-GPU tests and CI job for DDP validation

[ivanbasov]

Summary

• Add code/tests/test_multi_gpu.py with three short-running tests (skip unless torch.cuda.device_count() >= 2):
  • NCCL all_reduce: 2 ranks each hold (rank+1), verify sum = 3.0
  • DDP forward+backward: wraps PreDecoderModelMemory_v1 (d=3) in DDP, asserts all gradients finite after backward
  • Per-rank data generation: QCDataGeneratorTorch output lands on cuda:{rank}
• Add multi-gpu-tests job to ci-gpu.yml running on linux-amd64-gpu-rtxpro6000-latest-2 (post-merge only, 20 min timeout), with a 2-GPU DDP smoke train+inference and LER ≤ 0.35 check

Notes

• mp.spawn with file:// rendezvous avoids port conflicts between parallel test runs
• The smoke training step calls local_run.sh directly with GPUS=2 — smoke_run.sh hardcodes GPUS=1 so cannot be reused
• Runner label linux-amd64-gpu-rtxpro6000-latest-2 is assumed based on the existing -1 single-GPU naming pattern; confirm this label exists in the runner pool before merging, otherwise the job will queue indefinitely

Test plan

• Confirm linux-amd64-gpu-rtxpro6000-latest-2 runner label exists
• Verify test_multi_gpu.py skips cleanly on single-GPU CI runners (existing gpu-tests job)
• multi-gpu-tests job passes on a 2-GPU runner after merge to main

🤖 Generated with Claude Code

[bmhowe23]

Thank you, @ivanbasov. @kvmto - do you know if the tests in this PR would've caught the multi-GPU integration issue you previously mentioned today (with respect to cuStabilizer)?

[kvmto]

Thank you, @ivanbasov. @kvmto - do you know if the tests in this PR would've caught the multi-GPU integration issue you previously mentioned today (with respect to cuStabilizer)?

This overlaps with what I found. I would merge it.

[kvmto]

Thanks!