multiprocessing.spawn: allow a grace period when shutdown

[ppwwyyxx]

When one process fails, others are immediately killed. This prevents other processes to do necessary cleanups, or dump debug information (in particular, the NCCL flight recorder).

This PR adds a grace period. Default behavior is unchanged.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/131278

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 7fbfa5c with merge base ca38f28 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ppwwyyxx]

@albanD Hi Alban, could you take a look? thx

[albanD]

cc @wconstab from the distributed side and @andrewkho on the dataloader side. Any comment on this proposal?

[ppwwyyxx]

@wconstab any comments?

[albanD]

What's your strategy to test this?

[andrewkho]

Sorry for the delay in responding, in general the idea looks good to me, need to review the implementation more closely. i do agree with all of Alban's suggestions so please make those

[ppwwyyxx]

@pytorchbot label "release notes: distributed (miscellaneous)"

[ppwwyyxx]

Added a unittest and addressed @albanD 's comments.

[albanD]

Thanks!

[ppwwyyxx]

@pytorchbot merge

[pytorchmergebot]

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here