DDPOptimizer replace debug=True/False with using torchdynamo logger #88480

wconstab · 2022-11-04T05:10:33Z

Stack from ghstack (oldest at bottom):

Example output:

2022-11-04 05:09:29,525] torch._dynamo.optimizations.distributed: [INFO]
DDPOptimizer bucket assignments             
┌─────────┬────────────┬───────────────────┐       
│   Index │   Size (b) │ Param Names       │       
├─────────┼────────────┼───────────────────┤       
│       0 │  100120020 │ self_net_6_weight │       
├─────────┼────────────┼───────────────────┤       
│         │            │ self_net_6_bias   │       
├─────────┼────────────┼───────────────────┤       
│         │            │ self_net_4_weight │       
├─────────┼────────────┼───────────────────┤       
│         │            │ self_net_4_bias   │                              
├─────────┼────────────┼───────────────────┤
│       1 │  100020000 │ self_net_2_weight │
├─────────┼────────────┼───────────────────┤            
│         │            │ self_net_2_bias   │                                                          
├─────────┼────────────┼───────────────────┤
│       2 │     220000 │ self_net_0_weight │
├─────────┼────────────┼───────────────────┤
│         │            │ self_net_0_bias   │
└─────────┴────────────┴───────────────────┘
[2022-11-04 05:09:29,527] torch._dynamo.optimizations.distributed: [DEBUG]
---orig graph---
graph():
    %inputs : torch.Tensor [#users=1] = placeholder[target=inputs]
    %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {})
    %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {})
    %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {})
    %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {})
    %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {})
    %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {})
    %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {})
    %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {})
    return (self_net_7,)

---split graph---
graph():
    %inputs : torch.Tensor [#users=1] = placeholder[target=inputs]
    %submod_0 : [#users=1] = call_module[target=submod_0](args = (%inputs,), kwargs = {})
    %submod_1 : [#users=1] = call_module[target=submod_1](args = (%submod_0,), kwargs = {})
    %submod_2 : [#users=1] = call_module[target=submod_2](args = (%submod_1,), kwargs = {})
    return (submod_2,)

---submod_0 graph---
graph():
    %inputs : [#users=1] = placeholder[target=inputs]
    %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {})
    %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {})
    return self_net_1

---submod_1 graph---
graph():
    %self_net_1 : [#users=1] = placeholder[target=self_net_1]
    %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {})
    %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {})
    return self_net_3

---submod_2 graph---
graph():
    %self_net_3 : [#users=1] = placeholder[target=self_net_3]
    %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {})
    %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {})
    %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {})
    %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {})
    return self_net_7

---------------

cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @chunyuan-w @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx

[ghstack-poisoned]

pytorch-bot · 2022-11-04T05:10:37Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88480

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 Failures

As of commit 9988375:

The following jobs have failed:

cuda11.6-py3.10-gcc7-sm86 / test (default, 2, 4, linux.g5.4xlarge.nvidia.gpu)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 238abfa5ddae49712f3d1b22dda3b6ba05a46788 Pull Request resolved: #88480

…mo logger" Example output: ``` 2022-11-04 05:09:29,525] torch._dynamo.optimizations.distributed: [INFO] DDPOptimizer bucket assignments ┌─────────┬────────────┬───────────────────┐ │ Index │ Size (b) │ Param Names │ ├─────────┼────────────┼───────────────────┤ │ 0 │ 100120020 │ self_net_6_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_6_bias │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_4_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_4_bias │ ├─────────┼────────────┼───────────────────┤ │ 1 │ 100020000 │ self_net_2_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_2_bias │ ├─────────┼────────────┼───────────────────┤ │ 2 │ 220000 │ self_net_0_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_0_bias │ └─────────┴────────────┴───────────────────┘ [2022-11-04 05:09:29,527] torch._dynamo.optimizations.distributed: [DEBUG] ---orig graph--- graph(): %inputs : torch.Tensor [#users=1] = placeholder[target=inputs] %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {}) %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {}) %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {}) %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {}) %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {}) %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {}) %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {}) %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {}) return (self_net_7,) ---split graph--- graph(): %inputs : torch.Tensor [#users=1] = placeholder[target=inputs] %submod_0 : [#users=1] = call_module[target=submod_0](args = (%inputs,), kwargs = {}) %submod_1 : [#users=1] = call_module[target=submod_1](args = (%submod_0,), kwargs = {}) %submod_2 : [#users=1] = call_module[target=submod_2](args = (%submod_1,), kwargs = {}) return (submod_2,) ---submod_0 graph--- graph(): %inputs : [#users=1] = placeholder[target=inputs] %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {}) %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {}) return self_net_1 ---submod_1 graph--- graph(): %self_net_1 : [#users=1] = placeholder[target=self_net_1] %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {}) %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {}) return self_net_3 ---submod_2 graph--- graph(): %self_net_3 : [#users=1] = placeholder[target=self_net_3] %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {}) %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {}) %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {}) %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {}) return self_net_7 --------------- ``` cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen chunyuan-w XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

2022-11-04 05:09:29,525] torch._dynamo.optimizations.distributed: [INFO] DDPOptimizer bucket assignments ┌─────────┬────────────┬───────────────────┐ │ Index │ Size (b) │ Param Names │ ├─────────┼────────────┼───────────────────┤ │ 0 │ 100120020 │ self_net_6_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_6_bias │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_4_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_4_bias │ ├─────────┼────────────┼───────────────────┤ │ 1 │ 100020000 │ self_net_2_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_2_bias │ ├─────────┼────────────┼───────────────────┤ │ 2 │ 220000 │ self_net_0_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_0_bias │ └─────────┴────────────┴───────────────────┘ [2022-11-04 05:09:29,527] torch._dynamo.optimizations.distributed: [DEBUG] ---orig graph--- graph(): %inputs : torch.Tensor [#users=1] = placeholder[target=inputs] %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {}) %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {}) %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {}) %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {}) %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {}) %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {}) %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {}) %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {}) return (self_net_7,) ---split graph--- graph(): %inputs : torch.Tensor [#users=1] = placeholder[target=inputs] %submod_0 : [#users=1] = call_module[target=submod_0](args = (%inputs,), kwargs = {}) %submod_1 : [#users=1] = call_module[target=submod_1](args = (%submod_0,), kwargs = {}) %submod_2 : [#users=1] = call_module[target=submod_2](args = (%submod_1,), kwargs = {}) return (submod_2,) ---submod_0 graph--- graph(): %inputs : [#users=1] = placeholder[target=inputs] %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {}) %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {}) return self_net_1 ---submod_1 graph--- graph(): %self_net_1 : [#users=1] = placeholder[target=self_net_1] %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {}) %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {}) return self_net_3 ---submod_2 graph--- graph(): %self_net_3 : [#users=1] = placeholder[target=self_net_3] %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {}) %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {}) %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {}) %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {}) return self_net_7 --------------- ghstack-source-id: 4ee80a88972265e5de84945ca1f95808862e8af5 Pull Request resolved: #88480

…mo logger" Example output: ``` 2022-11-04 05:09:29,525] torch._dynamo.optimizations.distributed: [INFO] DDPOptimizer bucket assignments ┌─────────┬────────────┬───────────────────┐ │ Index │ Size (b) │ Param Names │ ├─────────┼────────────┼───────────────────┤ │ 0 │ 100120020 │ self_net_6_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_6_bias │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_4_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_4_bias │ ├─────────┼────────────┼───────────────────┤ │ 1 │ 100020000 │ self_net_2_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_2_bias │ ├─────────┼────────────┼───────────────────┤ │ 2 │ 220000 │ self_net_0_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_0_bias │ └─────────┴────────────┴───────────────────┘ [2022-11-04 05:09:29,527] torch._dynamo.optimizations.distributed: [DEBUG] ---orig graph--- graph(): %inputs : torch.Tensor [#users=1] = placeholder[target=inputs] %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {}) %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {}) %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {}) %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {}) %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {}) %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {}) %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {}) %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {}) return (self_net_7,) ---split graph--- graph(): %inputs : torch.Tensor [#users=1] = placeholder[target=inputs] %submod_0 : [#users=1] = call_module[target=submod_0](args = (%inputs,), kwargs = {}) %submod_1 : [#users=1] = call_module[target=submod_1](args = (%submod_0,), kwargs = {}) %submod_2 : [#users=1] = call_module[target=submod_2](args = (%submod_1,), kwargs = {}) return (submod_2,) ---submod_0 graph--- graph(): %inputs : [#users=1] = placeholder[target=inputs] %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {}) %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {}) return self_net_1 ---submod_1 graph--- graph(): %self_net_1 : [#users=1] = placeholder[target=self_net_1] %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {}) %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {}) return self_net_3 ---submod_2 graph--- graph(): %self_net_3 : [#users=1] = placeholder[target=self_net_3] %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {}) %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {}) %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {}) %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {}) return self_net_7 --------------- ``` cc mlazos soumith voznesenskym yanboliang penguinwu anijain2305 EikanWang jgong5 Guobing-Chen chunyuan-w XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx [ghstack-poisoned]

2022-11-04 05:09:29,525] torch._dynamo.optimizations.distributed: [INFO] DDPOptimizer bucket assignments ┌─────────┬────────────┬───────────────────┐ │ Index │ Size (b) │ Param Names │ ├─────────┼────────────┼───────────────────┤ │ 0 │ 100120020 │ self_net_6_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_6_bias │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_4_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_4_bias │ ├─────────┼────────────┼───────────────────┤ │ 1 │ 100020000 │ self_net_2_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_2_bias │ ├─────────┼────────────┼───────────────────┤ │ 2 │ 220000 │ self_net_0_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_0_bias │ └─────────┴────────────┴───────────────────┘ [2022-11-04 05:09:29,527] torch._dynamo.optimizations.distributed: [DEBUG] ---orig graph--- graph(): %inputs : torch.Tensor [#users=1] = placeholder[target=inputs] %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {}) %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {}) %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {}) %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {}) %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {}) %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {}) %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {}) %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {}) return (self_net_7,) ---split graph--- graph(): %inputs : torch.Tensor [#users=1] = placeholder[target=inputs] %submod_0 : [#users=1] = call_module[target=submod_0](args = (%inputs,), kwargs = {}) %submod_1 : [#users=1] = call_module[target=submod_1](args = (%submod_0,), kwargs = {}) %submod_2 : [#users=1] = call_module[target=submod_2](args = (%submod_1,), kwargs = {}) return (submod_2,) ---submod_0 graph--- graph(): %inputs : [#users=1] = placeholder[target=inputs] %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {}) %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {}) return self_net_1 ---submod_1 graph--- graph(): %self_net_1 : [#users=1] = placeholder[target=self_net_1] %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {}) %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {}) return self_net_3 ---submod_2 graph--- graph(): %self_net_3 : [#users=1] = placeholder[target=self_net_3] %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {}) %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {}) %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {}) %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {}) return self_net_7 --------------- ghstack-source-id: b952f2732720f17b08b6ee13bb4f38b553440c1c Pull Request resolved: #88480

anj-s

Nice!

wconstab · 2022-11-04T21:41:14Z

@pytorchbot merge

pytorchmergebot · 2022-11-04T21:43:06Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2022-11-04T22:18:27Z

Merge failed

Reason: 2 additional jobs have failed, first few of them are: trunk ,trunk / cuda11.6-py3.10-gcc7-sm86 / test (default, 2, 4, linux.g5.4xlarge.nvidia.gpu)

Details for Dev Infra team

Raised by workflow job

wconstab · 2022-11-05T02:35:46Z

@pytorchbot merge -f "Flaky CI"

pytorchmergebot · 2022-11-05T02:40:45Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…ytorch#88480) Example output: ``` 2022-11-04 05:09:29,525] torch._dynamo.optimizations.distributed: [INFO] DDPOptimizer bucket assignments ┌─────────┬────────────┬───────────────────┐ │ Index │ Size (b) │ Param Names │ ├─────────┼────────────┼───────────────────┤ │ 0 │ 100120020 │ self_net_6_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_6_bias │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_4_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_4_bias │ ├─────────┼────────────┼───────────────────┤ │ 1 │ 100020000 │ self_net_2_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_2_bias │ ├─────────┼────────────┼───────────────────┤ │ 2 │ 220000 │ self_net_0_weight │ ├─────────┼────────────┼───────────────────┤ │ │ │ self_net_0_bias │ └─────────┴────────────┴───────────────────┘ [2022-11-04 05:09:29,527] torch._dynamo.optimizations.distributed: [DEBUG] ---orig graph--- graph(): %inputs : torch.Tensor [#users=1] = placeholder[target=inputs] %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {}) %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {}) %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {}) %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {}) %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {}) %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {}) %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {}) %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {}) return (self_net_7,) ---split graph--- graph(): %inputs : torch.Tensor [#users=1] = placeholder[target=inputs] %submod_0 : [#users=1] = call_module[target=submod_0](args = (%inputs,), kwargs = {}) %submod_1 : [#users=1] = call_module[target=submod_1](args = (%submod_0,), kwargs = {}) %submod_2 : [#users=1] = call_module[target=submod_2](args = (%submod_1,), kwargs = {}) return (submod_2,) ---submod_0 graph--- graph(): %inputs : [#users=1] = placeholder[target=inputs] %self_net_0 : [#users=1] = call_module[target=self_net_0](args = (%inputs,), kwargs = {}) %self_net_1 : [#users=1] = call_module[target=self_net_1](args = (%self_net_0,), kwargs = {}) return self_net_1 ---submod_1 graph--- graph(): %self_net_1 : [#users=1] = placeholder[target=self_net_1] %self_net_2 : [#users=1] = call_module[target=self_net_2](args = (%self_net_1,), kwargs = {}) %self_net_3 : [#users=1] = call_module[target=self_net_3](args = (%self_net_2,), kwargs = {}) return self_net_3 ---submod_2 graph--- graph(): %self_net_3 : [#users=1] = placeholder[target=self_net_3] %self_net_4 : [#users=1] = call_module[target=self_net_4](args = (%self_net_3,), kwargs = {}) %self_net_5 : [#users=1] = call_module[target=self_net_5](args = (%self_net_4,), kwargs = {}) %self_net_6 : [#users=1] = call_module[target=self_net_6](args = (%self_net_5,), kwargs = {}) %self_net_7 : [#users=1] = call_module[target=self_net_7](args = (%self_net_6,), kwargs = {}) return self_net_7 --------------- ``` Pull Request resolved: pytorch#88480 Approved by: https://github.com/anj-s, https://github.com/davidberard98

DDPOptimizer replace debug=True/False with using torchdynamo logger

a2e8314

[ghstack-poisoned]

wconstab requested review from mrshenli, pritamdamania87, zhaojuanmao, rohan-varma, H-Huang, awgu and kwen2501 as code owners November 4, 2022 05:10

wconstab mentioned this pull request Nov 4, 2022

Support DDP ignored parameters in DDPOptimizer #88460

Closed

github-actions bot requested review from albanD, anjali411, antoniojkim, bdhirsh, Chillee, ezyang and miladm November 4, 2022 05:10

github-actions bot added ciflow/inductor module: dynamo labels Nov 4, 2022

wconstab added a commit that referenced this pull request Nov 4, 2022

DDPOptimizer replace debug=True/False with using torchdynamo logger

b85322d

ghstack-source-id: 238abfa5ddae49712f3d1b22dda3b6ba05a46788 Pull Request resolved: #88480

wconstab added the release notes: distributed (ddp) release notes category label Nov 4, 2022

anj-s approved these changes Nov 4, 2022

View reviewed changes

davidberard98 approved these changes Nov 4, 2022

View reviewed changes

wconstab mentioned this pull request Nov 4, 2022

Add single-process DDP accuracy support to dynamo benchmark suite #88511

Closed

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 4, 2022

wconstab mentioned this pull request Nov 4, 2022

Add docstring to DDPOptimizer #88521

Closed

wconstab mentioned this pull request Nov 4, 2022

Enable DDPOptimizer by default in dynamo #88523

Closed

pytorchmergebot added the Merged label Nov 5, 2022

pytorchmergebot closed this in 3fd0729 Nov 5, 2022

facebook-github-bot deleted the gh/wconstab/28/head branch June 8, 2023 19:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DDPOptimizer replace debug=True/False with using torchdynamo logger #88480

DDPOptimizer replace debug=True/False with using torchdynamo logger #88480

wconstab commented Nov 4, 2022 •

edited

pytorch-bot bot commented Nov 4, 2022 •

edited

anj-s left a comment

wconstab commented Nov 4, 2022

pytorchmergebot commented Nov 4, 2022

pytorchmergebot commented Nov 4, 2022

wconstab commented Nov 5, 2022

pytorchmergebot commented Nov 5, 2022

DDPOptimizer replace debug=True/False with using torchdynamo logger #88480

DDPOptimizer replace debug=True/False with using torchdynamo logger #88480

Conversation

wconstab commented Nov 4, 2022 • edited

pytorch-bot bot commented Nov 4, 2022 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88480

❌ 1 Failures

anj-s left a comment

Choose a reason for hiding this comment

wconstab commented Nov 4, 2022

pytorchmergebot commented Nov 4, 2022

Merge started

pytorchmergebot commented Nov 4, 2022

Merge failed

wconstab commented Nov 5, 2022

pytorchmergebot commented Nov 5, 2022

Merge started

wconstab commented Nov 4, 2022 •

edited

pytorch-bot bot commented Nov 4, 2022 •

edited