Distributed Tests failing but CI is green

[kshitij12345]

On latest main 94c9494, CI flow for distributed shows success https://github.com/Lightning-AI/lightning-thunder/runs/23172744261

But looking at the log, there are a few tests that have failed.

Sample

=================================== FAILURES ===================================
_ CompileDDPTest.test_fsdp_grad_parity_with_without_bucketing_executor_nvfuser_bucketing_block_zero2 _
/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_distributed.py:533: in wrapper
    self._join_processes(fn)
/usr/local/lib/python3.10/dist-packages/torch/testing/_internal/common_distributed.py:752: in _join_processes
    self._check_return_codes(elapsed_time)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

Link to log: https://dev.azure.com/Lightning-AI/lightning/_build/results?buildId=196660&view=logs&j=47e66f3c-897a-5428-da11-bf5c7745762e&t=97be8351-284a-5dba-49eb-f9fe7c3ed1a2&l=811

cc @Borda

[kshitij12345]

Raw Log (in case the CI logs are cleaned): log.txt

[Borda]

Raw Log (in case the CI logs are cleaned): log.txt

so CI is fixed in #99 but probably we would also need to fix the issue... 🤔