[pipelining] [BE] Move pipeline_order validation to schedules.py #129369

H-Huang · 2024-06-24T14:00:36Z

Stack from ghstack (oldest at bottom):

Changes

small fix in stage error message
Move format_pipeline_order and _validate_pipeline_order out of test_schedule.py into schedules.py.
Wrap the execution runtime in a try-except which on error will log the timestep and schedule plan before re-raising the exception.

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @chauhang @d4l3k

[ghstack-poisoned]

pytorch-bot · 2024-06-24T14:00:40Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129369

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

Checkout action fails due to incompatible GLIBC

✅ No Failures

As of commit 57f195c with merge base 37e3c60 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: bea3bd8 Pull Request resolved: #129369

# Changes * Move `format_pipeline_order` and `_validate_pipeline_order` out of `test_schedule.py` into `schedules.py`. * Wrap the execution runtime in a try-except which on error will log the timestep and schedule plan before re-raising the exception. cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k [ghstack-poisoned]

# Changes * small fix in stage error message * Move `format_pipeline_order` and `_validate_pipeline_order` out of `test_schedule.py` into `schedules.py`. * Wrap the execution runtime in a try-except which on error will log the timestep and schedule plan before re-raising the exception. cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k [ghstack-poisoned]

ghstack-source-id: 49b786f Pull Request resolved: #129369

wconstab · 2024-06-28T17:33:21Z

torch/distributed/pipelining/schedules.py

+    ]
+    # Join the rows into a single string
+    formatted_table = (
+        "=========== ALL_RANK_ACTIONS ===========\n"


nit: i think this function could stay as a helper function maybe called by repr

and it's a little better IMO to remove the ====ALL_RANK.. part so that inside repr we can customize a bit more

repr should probably print out the class name and relevant field values, and maybe print out some shortened summary of the actions but not the whole page-long actions by default? I'd suggest looking at hohw torch.tensor repr works

PipelineSchedule
[..possibly, some args such as number of stages, etc]
Actions
[ ... ]

wconstab · 2024-06-28T17:37:18Z

torch/distributed/pipelining/schedules.py

+                    _batch_p2p(ops).wait()
+            except Exception as e:
+                logger.error(
+                    "Exception in rank %s at time step %s", self.rank, time_step


this string should include more info. not sure what amount is right but some things that might be good

which stage?

which action?

Maybe also phrase it more like "PipelineSchedule {schedulename?} caught the following exception when running {}th microbatch {Action} for stage {}" or something?

…les.py" # Changes * small fix in stage error message * Move `format_pipeline_order` and `_validate_pipeline_order` out of `test_schedule.py` into `schedules.py`. * Wrap the execution runtime in a try-except which on error will log the timestep and schedule plan before re-raising the exception. cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k [ghstack-poisoned]

H-Huang · 2024-07-01T14:27:45Z

@pytorchbot merge

pytorchmergebot · 2024-07-01T14:29:32Z

Merge failed

Reason: This PR needs a release notes: label
If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Details for Dev Infra team

Raised by workflow job

…les.py" # Changes * small fix in stage error message * Move `format_pipeline_order` and `_validate_pipeline_order` out of `test_schedule.py` into `schedules.py`. * Wrap the execution runtime in a try-except which on error will log the timestep and schedule plan before re-raising the exception. cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k [ghstack-poisoned]

ghstack-source-id: 1e7866e Pull Request resolved: #129369

H-Huang · 2024-07-02T13:07:03Z

@pytorchbot merge -i

…les.py" # Changes * small fix in stage error message * Move `format_pipeline_order` and `_validate_pipeline_order` out of `test_schedule.py` into `schedules.py`. * Wrap the execution runtime in a try-except which on error will log the timestep and schedule plan before re-raising the exception. cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k [ghstack-poisoned]

ghstack-source-id: bccf0da Pull Request resolved: #129369

H-Huang · 2024-07-02T18:12:18Z

@pytorchbot merge -i

pytorchmergebot · 2024-07-02T18:13:50Z

Merge started

Your change will be merged while ignoring the following 2 checks: pull / linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu), trunk / linux-focal-cuda11.8-py3.10-gcc9-experimental-split-build-test / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

clee2000 · 2024-07-02T22:29:09Z

@pytorchbot revert -m "broke test/distributed/pipelining/test_schedule.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass0 on distributed cuda https://github.com/pytorch/pytorch/actions/runs/9766039400/job/26959115773 https://hud.pytorch.org/pytorch/pytorch/commit/ec789a3c9ddd4e550b3dea6934ce2d41deb98784. You can see the error on the PR, but Dr. CI classified it wrong" -c weird

pytorchmergebot · 2024-07-02T22:30:44Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2024-07-02T22:30:55Z

@H-Huang your PR has been successfully reverted.

….py (#129369)" This reverts commit ec789a3. Reverted #129369 on behalf of https://github.com/clee2000 due to broke test/distributed/pipelining/test_schedule.py::ScheduleTest::test_non_symmetric_stage_ids_ScheduleClass0 on distributed cuda https://github.com/pytorch/pytorch/actions/runs/9766039400/job/26959115773 https://hud.pytorch.org/pytorch/pytorch/commit/ec789a3c9ddd4e550b3dea6934ce2d41deb98784. You can see the error on the PR, but Dr. CI classified it wrong ([comment](#129369 (comment)))

…les.py" # Changes * small fix in stage error message * Move `format_pipeline_order` and `_validate_pipeline_order` out of `test_schedule.py` into `schedules.py`. * Wrap the execution runtime in a try-except which on error will log the timestep and schedule plan before re-raising the exception. cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k [ghstack-poisoned]

ghstack-source-id: affdd01 Pull Request resolved: #129369

H-Huang · 2024-07-03T18:49:37Z

Thanks for catching that @clee2000, yeah the error was legit. Fixed now

H-Huang · 2024-07-03T18:50:29Z

@pytorchbot merge

pytorchmergebot · 2024-07-03T18:51:57Z

Merge failed

Reason: Not merging any PRs at the moment because there is a merge blocking https://github.com/pytorch/pytorch/labels/ci:%20sev issue open at:
#130041

Details for Dev Infra team

Raised by workflow job

H-Huang · 2024-07-04T16:31:31Z

@pytorchbot merge

pytorchmergebot · 2024-07-04T16:33:08Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

[pipelining] [BE] Move pipeline_order validation to utils

93f2ee4

[ghstack-poisoned]

H-Huang mentioned this pull request Jun 24, 2024

[pipelining] add small logging section to docs #129368

Closed

pytorch-bot bot added the oncall: distributed Add this issue/PR to distributed oncall triage queue label Jun 24, 2024

H-Huang added a commit that referenced this pull request Jun 24, 2024

[pipelining] [BE] Move pipeline_order validation to utils

2be4bcb

ghstack-source-id: bea3bd8 Pull Request resolved: #129369

H-Huang requested review from wconstab June 24, 2024 14:11

H-Huang added a commit that referenced this pull request Jun 25, 2024

[pipelining] [BE] Move pipeline_order validation to utils

c139da3

ghstack-source-id: 49b786f Pull Request resolved: #129369

H-Huang changed the title ~~[pipelining] [BE] Move pipeline_order validation to utils~~ [pipelining] [BE] Move pipeline_order validation to schedules.py Jun 26, 2024

H-Huang requested a review from haocizhang June 26, 2024 19:03

wconstab reviewed Jun 28, 2024

View reviewed changes

wconstab approved these changes Jun 28, 2024

View reviewed changes

wconstab reviewed Jun 28, 2024

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 1, 2024

pytorchmergebot added the merging label Jul 1, 2024

pytorchmergebot removed the merging label Jul 1, 2024

H-Huang added the release notes: distributed (pipeline) release notes category label Jul 1, 2024

H-Huang added a commit that referenced this pull request Jul 1, 2024

[pipelining] [BE] Move pipeline_order validation to utils

e6b2745

ghstack-source-id: 1e7866e Pull Request resolved: #129369

pytorchmergebot added the merging label Jul 2, 2024

H-Huang added a commit that referenced this pull request Jul 2, 2024

[pipelining] [BE] Move pipeline_order validation to utils

e18cf26

ghstack-source-id: bccf0da Pull Request resolved: #129369

pytorchmergebot added the merging label Jul 2, 2024

pytorchmergebot added the Merged label Jul 2, 2024

pytorchmergebot closed this in ec789a3 Jul 2, 2024

pytorchmergebot removed the merging label Jul 2, 2024

pytorchmergebot added the Reverted label Jul 2, 2024

pytorchmergebot reopened this Jul 2, 2024

H-Huang added a commit that referenced this pull request Jul 3, 2024

[pipelining] [BE] Move pipeline_order validation to utils

30521dd

ghstack-source-id: affdd01 Pull Request resolved: #129369

pytorchmergebot added the merging label Jul 3, 2024

pytorchmergebot removed the merging label Jul 3, 2024

pytorchmergebot added the merging label Jul 4, 2024

pytorchmergebot closed this in 5b5f4b0 Jul 4, 2024

pytorchmergebot removed the merging label Jul 4, 2024

github-actions bot deleted the gh/H-Huang/130/head branch August 4, 2024 02:02

[pipelining] [BE] Move pipeline_order validation to schedules.py #129369

[pipelining] [BE] Move pipeline_order validation to schedules.py #129369

Uh oh!

Conversation

H-Huang commented Jun 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

pytorch-bot bot commented Jun 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129369

❗ 1 Active SEVs

✅ No Failures

Uh oh!

wconstab Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

wconstab Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

H-Huang commented Jul 1, 2024

Uh oh!

pytorchmergebot commented Jul 1, 2024

Merge failed

Uh oh!

H-Huang commented Jul 2, 2024

Uh oh!

H-Huang commented Jul 2, 2024

Uh oh!

pytorchmergebot commented Jul 2, 2024

Merge started

Uh oh!

clee2000 commented Jul 2, 2024

Uh oh!

pytorchmergebot commented Jul 2, 2024

Uh oh!

pytorchmergebot commented Jul 2, 2024

Uh oh!

H-Huang commented Jul 3, 2024

Uh oh!

H-Huang commented Jul 3, 2024

Uh oh!

pytorchmergebot commented Jul 3, 2024

Merge failed

Uh oh!

H-Huang commented Jul 4, 2024

Uh oh!

pytorchmergebot commented Jul 4, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

H-Huang commented Jun 24, 2024 •

edited

Loading

pytorch-bot bot commented Jun 24, 2024 •

edited

Loading