Do Not Merge: set watchdog timeout to 1ms

[booxter]

I expect it to fail training, proving the knob works.

Signed-off-by: Ihar Hrachyshka ihar.hrachyshka@gmail.com

[github-actions]

E2E (NVIDIA L40S x4) workflow launched on this PR: View run

[github-actions]

e2e workflow failed on this PR: View run, please investigate.

[booxter]

terminate called after throwing an instance of 'c10::DistBackendError'
  what():  [PG ID 0 PG GUID 0(default_pg) Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, NumelIn=1, NumelOut=1, Timeout(ms)=1) ran for 90 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f83f296c1b6 in /actions-runner/_work/training/training/instructlab/venv/lib64/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x2b4 (0x7f83a09fec74 in /actions-runner/_work/training/training/instructlab/venv/lib64/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x890 (0x7f83a0a007d0 in /actions-runner/_work/training/training/instructlab/venv/lib64/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f83a0a016ed in /actions-runner/_work/training/training/instructlab/venv/lib64/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x145c0 (0x7f83f33ac5c0 in /actions-runner/_work/training/training/instructlab/venv/lib64/python3.11/site-packages/torch/lib/libtorch.so)
frame #5: <unknown function> + 0x89c02 (0x7f83f3c89c02 in /lib64/libc.so.6)
frame #6: <unknown function> + 0x10ec40 (0x7f83f3d0ec40 in /lib64/libc.so.6)

Worked like a charm.