Split DeepCompile ZeRO-3 memory scheduler - #8233
Conversation
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 255d4347ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| int(get_accelerator().max_memory_allocated()) + int(mem_usage_out_of_torch)) | ||
|
|
||
|
|
||
| def _rank_max_profile_memory(start_mem, peak_mem, device, distributed, process_group=None): |
There was a problem hiding this comment.
When an operator fails on only one data-parallel rank—for example, due to rank-asymmetric memory pressure—that rank exits ProfilingInterpreter.run_node, while successful ranks enter this per-node MAX reduction. The failed rank then advances to cleanup and later collectives, so the ranks execute different collective sequences and the distributed compilation can hang instead of marking the profile incomplete. Avoid entering the per-node reduction until profiling success or failure has been coordinated across the process group.
Useful? React with 👍 / 👎.
|
|
||
| # Appendix | ||
|
|
||
| ## Diagnostics |
There was a problem hiding this comment.
Commit 03008a7b862ba99e88940209267d6e0a82f60267 is a non-merge commit but its message has no Signed-off-by trailer, so it violates the repository's commit requirement and will fail DCO-style validation; recreate the commit with --signoff.
AGENTS.md reference: AGENTS.md:L8-L8
Useful? React with 👍 / 👎.
1 participant
Splits the DeepCompile ZeRO-3 scheduling and gather/release lifetime changes out of #8169. The exact candidate passed 10/10 tests on a single node with 2 H100 GPUs.