[PT-D][checkpoint] Resolve no such file or directory issue when checkpointing on multi hosts #92553

wz337 · 2023-01-18T17:13:10Z

Previously, we only create the directory in rank 0. Therefore, if running on multihosts with multiple GPUs, we would run into issues of "No such file or directory".

This is the fix for it.

pytorch-bot · 2023-01-18T17:13:13Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/92553

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 998f358:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

kumpera · 2023-01-18T17:26:56Z

torch/distributed/checkpoint/filesystem.py

@@ -349,7 +349,7 @@ def init(self, is_coordinator: bool) -> None:
        pass

    def prepare_local_plan(self, plan: SavePlan) -> SavePlan:
-        # There's no storage input in the local plan
+        os.makedirs(self.path, exist_ok=True)


Looks good, but we should remove from prepare_global_plan as it's now redundant.

kumpera

LGTM

wz337 · 2023-01-18T19:06:41Z

@pytorchmergebot merge

pytorchmergebot · 2023-01-18T19:08:24Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2023-01-18T19:33:43Z

Merge failed

Reason: The following mandatory check(s) failed (Rule Distributed):

pull

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

wz337 · 2023-01-18T20:03:31Z

@pytorchmergebot rebase

pytorchmergebot · 2023-01-18T20:05:32Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2023-01-18T20:05:37Z

Successfully rebased multi_host_filesystemwriter onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout multi_host_filesystemwriter && git pull --rebase)

wz337 · 2023-01-18T21:13:14Z

@pytorchmergebot merge

pytorchmergebot · 2023-01-18T21:14:56Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2023-01-18T22:20:40Z

Merge failed

Reason: The following mandatory check(s) failed (Rule Distributed):

pull

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

wz337 · 2023-01-19T04:28:40Z

@pytorchmergebot rebase

pytorchmergebot · 2023-01-19T04:30:44Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2023-01-19T04:30:49Z

Successfully rebased multi_host_filesystemwriter onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout multi_host_filesystemwriter && git pull --rebase)

wz337 · 2023-01-19T04:43:45Z

@pytorchmergebot merge

pytorchmergebot · 2023-01-19T04:45:27Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2023-01-19T10:44:10Z

The merge job was canceled. If you believe this is a mistake,then you can re trigger it through pytorch-bot.

wz337 · 2023-01-19T16:40:28Z

@pytorchmergebot rebase

pytorchmergebot · 2023-01-19T16:42:21Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2023-01-19T16:42:26Z

Successfully rebased multi_host_filesystemwriter onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout multi_host_filesystemwriter && git pull --rebase)

wz337 · 2023-01-20T18:07:37Z

@pytorchmergebot rebase

pytorchmergebot · 2023-01-20T18:09:29Z

@pytorchbot successfully started a rebase job. Check the current status here

pytorchmergebot · 2023-01-20T18:09:34Z

Successfully rebased multi_host_filesystemwriter onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout multi_host_filesystemwriter && git pull --rebase)

wz337 · 2023-01-20T20:16:16Z

@pytorchmergebot merge

pytorchmergebot · 2023-01-20T20:17:56Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

wz337 marked this pull request as ready for review January 18, 2023 17:13

wz337 requested review from mrshenli, zhaojuanmao, rohan-varma, H-Huang, awgu, kwen2501 and wanchaol as code owners January 18, 2023 17:13

wz337 requested a review from kumpera January 18, 2023 17:13

kumpera suggested changes Jan 18, 2023

View reviewed changes

wz337 requested a review from kumpera January 18, 2023 19:00

kumpera approved these changes Jan 18, 2023

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jan 18, 2023

wz337 added 3 commits January 20, 2023 18:09

add os.makedirs()

7b9393e

update

2a38d47

update prepare_local_plan and prepare_global_plan

998f358

pytorchmergebot added the Merged label Jan 20, 2023

pytorchmergebot closed this in eee2869 Jan 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PT-D][checkpoint] Resolve no such file or directory issue when checkpointing on multi hosts #92553

[PT-D][checkpoint] Resolve no such file or directory issue when checkpointing on multi hosts #92553

wz337 commented Jan 18, 2023 •

edited

pytorch-bot bot commented Jan 18, 2023 •

edited

kumpera Jan 18, 2023

kumpera left a comment

wz337 commented Jan 18, 2023

pytorchmergebot commented Jan 18, 2023

pytorchmergebot commented Jan 18, 2023

wz337 commented Jan 18, 2023

pytorchmergebot commented Jan 18, 2023

pytorchmergebot commented Jan 18, 2023

wz337 commented Jan 18, 2023

pytorchmergebot commented Jan 18, 2023

pytorchmergebot commented Jan 18, 2023

wz337 commented Jan 19, 2023

pytorchmergebot commented Jan 19, 2023

pytorchmergebot commented Jan 19, 2023

wz337 commented Jan 19, 2023

pytorchmergebot commented Jan 19, 2023

pytorchmergebot commented Jan 19, 2023

wz337 commented Jan 19, 2023

pytorchmergebot commented Jan 19, 2023

pytorchmergebot commented Jan 19, 2023

wz337 commented Jan 20, 2023

pytorchmergebot commented Jan 20, 2023

pytorchmergebot commented Jan 20, 2023

wz337 commented Jan 20, 2023

pytorchmergebot commented Jan 20, 2023

[PT-D][checkpoint] Resolve no such file or directory issue when checkpointing on multi hosts #92553

[PT-D][checkpoint] Resolve no such file or directory issue when checkpointing on multi hosts #92553

Conversation

wz337 commented Jan 18, 2023 • edited

pytorch-bot bot commented Jan 18, 2023 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/92553

✅ No Failures

kumpera Jan 18, 2023

Choose a reason for hiding this comment

kumpera left a comment

Choose a reason for hiding this comment

wz337 commented Jan 18, 2023

pytorchmergebot commented Jan 18, 2023

Merge started

pytorchmergebot commented Jan 18, 2023

Merge failed

wz337 commented Jan 18, 2023

pytorchmergebot commented Jan 18, 2023

pytorchmergebot commented Jan 18, 2023

wz337 commented Jan 18, 2023

pytorchmergebot commented Jan 18, 2023

Merge started

pytorchmergebot commented Jan 18, 2023

Merge failed

wz337 commented Jan 19, 2023

pytorchmergebot commented Jan 19, 2023

pytorchmergebot commented Jan 19, 2023

wz337 commented Jan 19, 2023

pytorchmergebot commented Jan 19, 2023

Merge started

pytorchmergebot commented Jan 19, 2023

wz337 commented Jan 19, 2023

pytorchmergebot commented Jan 19, 2023

pytorchmergebot commented Jan 19, 2023

wz337 commented Jan 20, 2023

pytorchmergebot commented Jan 20, 2023

pytorchmergebot commented Jan 20, 2023

wz337 commented Jan 20, 2023

pytorchmergebot commented Jan 20, 2023

Merge started

wz337 commented Jan 18, 2023 •

edited

pytorch-bot bot commented Jan 18, 2023 •

edited