Benchmark combining Distributed Data Parallel and Distributed RPC #46993

BorisValkov5 · 2020-10-28T16:25:50Z

Summary:
Introducing benchmark that combines Distributed Data Parallelism with Distributed Model Parallelism. The benchmark measures distributed training iteration time. The number of trainer nodes and parameter servers are configurable. The default setup has 8 trainers, 1 master node and 8 parameter servers.

The training process is executed as follows:

The master creates embedding tables on each of the 8 Parameter Servers and holds an RRef to it.
The master, then kicks off the training loop on the 8 trainers and passes the embedding table RRef to the trainers.
The trainers create a HybridModel which performs embedding lookups in all 8 Parameter Servers using the embedding table RRef provided by the master and then executes the FC layer which is wrapped and replicated via DDP (DistributedDataParallel).
The trainer executes the forward pass of the model and uses the loss to
execute the backward pass using Distributed Autograd.
As part of the backward pass, the gradients for the FC layer are computed
first and synced to all trainers via allreduce in DDP.
Next, Distributed Autograd propagates the gradients to the parameter servers,
where the gradients for the embedding table are updated.
Finally, the Distributed Optimizer is used to update all parameters.

Test Plan:
waitforbuildbot

Benchmark output:

---------- Info ---------

PyTorch version: 1.7.0
CUDA version: 9.2.0

---------- nvidia-smi topo -m ---------

GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU     Affinity
GPU0     X      NV2     NV1     NV2     NV1     NODE    NODE    NODE    0-19,40-59
GPU1    NV2      X      NV2     NV1     NODE    NV1     NODE    NODE    0-19,40-59
GPU2    NV1     NV2      X      NV1     NODE    NODE    NV2     NODE    0-19,40-59
GPU3    NV2     NV1     NV1      X      NODE    NODE    NODE    NV2     0-19,40-59
GPU4    NV1     NODE    NODE    NODE     X      NV2     NV1     NV2     0-19,40-59
GPU5    NODE    NV1     NODE    NODE    NV2      X      NV2     NV1     0-19,40-59
GPU6    NODE    NODE    NV2     NODE    NV1     NV2      X      NV1     0-19,40-59
GPU7    NODE    NODE    NODE    NV2     NV2     NV1     NV1      X      0-19,40-59

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
PIX = Connection traversing a single PCIe switch
NV# = Connection traversing a bonded set of # NVLinks

------------------ PyTorch Distributed Benchmark (DDP and RPC) ---------------------

                sec/iter    ex/sec      sec/iter    ex/sec      sec/iter    ex/sec      sec/iter    ex/sec
Trainer0:  p50:  0.376s     185/s  p75:  0.384s     182/s  p90:  0.390s     179/s  p95:  0.396s     176/s
Trainer1:  p50:  0.377s     204/s  p75:  0.384s     200/s  p90:  0.389s     197/s  p95:  0.393s     195/s
Trainer2:  p50:  0.377s     175/s  p75:  0.384s     172/s  p90:  0.390s     169/s  p95:  0.395s     166/s
Trainer3:  p50:  0.377s     161/s  p75:  0.384s     158/s  p90:  0.390s     156/s  p95:  0.393s     155/s
Trainer4:  p50:  0.377s     172/s  p75:  0.383s     169/s  p90:  0.389s     166/s  p95:  0.395s     164/s
Trainer5:  p50:  0.377s     180/s  p75:  0.383s     177/s  p90:  0.389s     174/s  p95:  0.395s     172/s
Trainer6:  p50:  0.377s     204/s  p75:  0.384s     200/s  p90:  0.390s     197/s  p95:  0.394s     195/s
Trainer7:  p50:  0.377s     185/s  p75:  0.384s     182/s  p90:  0.389s     179/s  p95:  0.394s     177/s
     All:  p50:  0.377s    1470/s  p75:  0.384s    1443/s  p90:  0.390s    1421/s  p95:  0.396s    1398/s

Differential Revision: D24409230

facebook-github-bot · 2020-10-28T16:26:25Z

This pull request was exported from Phabricator. Differential Revision: D24409230

dr-ci · 2020-10-28T17:01:09Z

💊 CI failures summary and remediations

As of commit 9d75306 (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 4 times.

pritamdamania87

Awesome stuff, the overall benchmark looks great! I have a few minor comments inline.

pritamdamania87 · 2020-10-29T01:00:02Z

torch/distributed/benchmarks/README.md

@@ -0,0 +1,68 @@
+# Benchmark combining Distributed Data Parallel and Distributed RPC


nit: since we might add other files to the benchmark directory, maybe we should create a ddp_rpc directory under this folder and put the README and the benchmark file under that folder.

pritamdamania87 · 2020-10-29T01:01:45Z

torch/distributed/benchmarks/README.md

+7) Finally, the [Distributed Optimizer](https://pytorch.org/docs/master/rpc.html#module-torch.distributed.optim) is used to update all parameters.
+
+
+## Example Benchmark output:


Probably add an example here of how to actually run the benchmark?

pritamdamania87 · 2020-10-29T01:06:18Z

torch/distributed/benchmarks/benchmark_ddp_rpc.py

+
+    measurements = []
+    # Include warm-up cycles during training
+    for epoch in range(100 + WARMUP_CYCLES):


How long does this benchmark take to run? I'd suggest maybe 1000 iterations and 100 WARMUP_CYCLES as the default if it doesn't take too long to run.

pritamdamania87 · 2020-10-29T01:07:26Z

torch/distributed/benchmarks/README.md

+
+------------------  PyTorch Distributed Benchmark (DDP and RPC) ---------------------
+
+                    sec/iter    ex/sec      sec/iter    ex/sec      sec/iter    ex/sec      sec/iter    ex/sec


These should be secs/epoch right and epochs/sec right?

pritamdamania87 · 2020-10-29T01:09:25Z

torch/distributed/benchmarks/benchmark_ddp_rpc.py

+        super(HybridModel, self).__init__()
+        self.emb_rref_list = emb_rref_list
+        self.fc = DDP(
+            torch.nn.Linear(NUM_PS * EMBEDDING_DIM, 8).cuda(device), device_ids=[device]


It would be better to have Linear layers of constant size and then reshape the output from torch.cat to match the first linear layer's shape.

pritamdamania87 · 2020-10-29T01:10:05Z

torch/distributed/benchmarks/benchmark_ddp_rpc.py

+        super(HybridModel, self).__init__()
+        self.emb_rref_list = emb_rref_list
+        self.fc = DDP(
+            torch.nn.Linear(NUM_PS * EMBEDDING_DIM, 8).cuda(device), device_ids=[device]


We should also have maybe 3-5 linear layers stacked one after the other to simulate a MLP.

pritamdamania87 · 2020-10-29T01:11:33Z

torch/distributed/benchmarks/benchmark_ddp_rpc.py

+NUM_EMBEDDINGS = 100
+EMBEDDING_DIM = 16


Maybe keep the default as 1000000 x 64 since this is a benchmark and not a test.

…torch#46993) Summary: Pull Request resolved: pytorch#46993 Introducing benchmark that combines Distributed Data Parallelism with Distributed Model Parallelism. The benchmark measures distributed training iteration time. The number of trainer nodes and parameter servers are configurable. The default setup has 8 trainers, 1 master node and 8 parameter servers. The training process is executed as follows: 1) The master creates embedding tables on each of the 8 Parameter Servers and holds an RRef to it. 2) The master, then kicks off the training loop on the 8 trainers and passes the embedding table RRef to the trainers. 3) The trainers create a `HybridModel` which performs embedding lookups in all 8 Parameter Servers using the embedding table RRef provided by the master and then executes the FC layer which is wrapped and replicated via DDP (DistributedDataParallel). 4) The trainer executes the forward pass of the model and uses the loss to execute the backward pass using Distributed Autograd. 5) As part of the backward pass, the gradients for the FC layer are computed first and synced to all trainers via allreduce in DDP. 6) Next, Distributed Autograd propagates the gradients to the parameter servers, where the gradients for the embedding table are updated. 7) Finally, the Distributed Optimizer is used to update all parameters. Test Plan: waitforbuildbot Benchmark output: ---------- Info --------- * PyTorch version: 1.7.0 * CUDA version: 9.2.0 ---------- nvidia-smi topo -m --------- GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 CPU Affinity GPU0 X NV2 NV1 NV2 NV1 NODE NODE NODE 0-19,40-59 GPU1 NV2 X NV2 NV1 NODE NV1 NODE NODE 0-19,40-59 GPU2 NV1 NV2 X NV1 NODE NODE NV2 NODE 0-19,40-59 GPU3 NV2 NV1 NV1 X NODE NODE NODE NV2 0-19,40-59 GPU4 NV1 NODE NODE NODE X NV2 NV1 NV2 0-19,40-59 GPU5 NODE NV1 NODE NODE NV2 X NV2 NV1 0-19,40-59 GPU6 NODE NODE NV2 NODE NV1 NV2 X NV1 0-19,40-59 GPU7 NODE NODE NODE NV2 NV2 NV1 NV1 X 0-19,40-59 Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge) PIX = Connection traversing a single PCIe switch NV# = Connection traversing a bonded set of # NVLinks ------------------ PyTorch Distributed Benchmark (DDP and RPC) --------------------- sec/iter ex/sec sec/iter ex/sec sec/iter ex/sec sec/iter ex/sec Trainer0: p50: 0.376s 185/s p75: 0.384s 182/s p90: 0.390s 179/s p95: 0.396s 176/s Trainer1: p50: 0.377s 204/s p75: 0.384s 200/s p90: 0.389s 197/s p95: 0.393s 195/s Trainer2: p50: 0.377s 175/s p75: 0.384s 172/s p90: 0.390s 169/s p95: 0.395s 166/s Trainer3: p50: 0.377s 161/s p75: 0.384s 158/s p90: 0.390s 156/s p95: 0.393s 155/s Trainer4: p50: 0.377s 172/s p75: 0.383s 169/s p90: 0.389s 166/s p95: 0.395s 164/s Trainer5: p50: 0.377s 180/s p75: 0.383s 177/s p90: 0.389s 174/s p95: 0.395s 172/s Trainer6: p50: 0.377s 204/s p75: 0.384s 200/s p90: 0.390s 197/s p95: 0.394s 195/s Trainer7: p50: 0.377s 185/s p75: 0.384s 182/s p90: 0.389s 179/s p95: 0.394s 177/s All: p50: 0.377s 1470/s p75: 0.384s 1443/s p90: 0.390s 1421/s p95: 0.396s 1398/s Differential Revision: D24409230 fbshipit-source-id: a9da9cfec013af6209f036dccca0361cc9305b0c

facebook-github-bot · 2020-11-03T19:20:23Z

This pull request was exported from Phabricator. Differential Revision: D24409230

codecov · 2020-11-03T22:50:33Z

Codecov Report

Merging #46993 into master will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #46993   +/-   ##
=======================================
  Coverage   60.81%   60.81%           
=======================================
  Files        2749     2749           
  Lines      254098   254098           
=======================================
+ Hits       154522   154533   +11     
+ Misses      99576    99565   -11

mrshenli

Hey @BorisValkov5, thanks for adding this. I have two more comments:

Shall we combine this torch/distributed/benchmark folder with benchmarks/distributed/ folder? It might be a little confusing to have two different benchmark folders.
Is it possible to add tests for benchmarks, so that CI can help us make sure that our benchmarks won't be broken by future changes.

BorisValkov5 · 2020-11-04T22:37:25Z

Thanks @mrshenli for your comments! Land in progress - will address them in a subsequent change.

facebook-github-bot · 2020-11-05T03:19:56Z

This pull request has been merged in 6c5a1c5.

BorisValkov5 requested review from apaszke, mingzhe09088, mrshenli, pietern, pritamdamania87, rohan-varma and zhaojuanmao as code owners October 28, 2020 16:25

facebook-github-bot added oncall: distributed Add this issue/PR to distributed oncall triage queue fb-exported labels Oct 28, 2020

pritamdamania87 reviewed Oct 29, 2020

View reviewed changes

facebook-github-bot added the cla signed label Oct 30, 2020

BorisValkov5 force-pushed the export-D24409230 branch from 5fe4158 to 9d75306 Compare November 3, 2020 19:20

pritamdamania87 approved these changes Nov 4, 2020

View reviewed changes

mrshenli reviewed Nov 4, 2020

View reviewed changes

facebook-github-bot closed this in 6c5a1c5 Nov 5, 2020

facebook-github-bot added the Merged label Nov 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmark combining Distributed Data Parallel and Distributed RPC #46993

Benchmark combining Distributed Data Parallel and Distributed RPC #46993

BorisValkov5 commented Oct 28, 2020

facebook-github-bot commented Oct 28, 2020

dr-ci bot commented Oct 28, 2020 •

edited

pritamdamania87 left a comment

pritamdamania87 Oct 29, 2020

pritamdamania87 Oct 29, 2020

pritamdamania87 Oct 29, 2020

pritamdamania87 Oct 29, 2020

pritamdamania87 Oct 29, 2020

pritamdamania87 Oct 29, 2020

pritamdamania87 Oct 29, 2020

facebook-github-bot commented Nov 3, 2020

codecov bot commented Nov 3, 2020

mrshenli left a comment

BorisValkov5 commented Nov 4, 2020

facebook-github-bot commented Nov 5, 2020

		@@ -0,0 +1,68 @@
		# Benchmark combining Distributed Data Parallel and Distributed RPC

		7) Finally, the [Distributed Optimizer](https://pytorch.org/docs/master/rpc.html#module-torch.distributed.optim) is used to update all parameters.


		## Example Benchmark output:


		------------------ PyTorch Distributed Benchmark (DDP and RPC) ---------------------

		sec/iter ex/sec sec/iter ex/sec sec/iter ex/sec sec/iter ex/sec

		NUM_EMBEDDINGS = 100
		EMBEDDING_DIM = 16

Benchmark combining Distributed Data Parallel and Distributed RPC #46993

Benchmark combining Distributed Data Parallel and Distributed RPC #46993

Conversation

BorisValkov5 commented Oct 28, 2020

facebook-github-bot commented Oct 28, 2020

dr-ci bot commented Oct 28, 2020 • edited

💊 CI failures summary and remediations

pritamdamania87 left a comment

Choose a reason for hiding this comment

pritamdamania87 Oct 29, 2020

Choose a reason for hiding this comment

pritamdamania87 Oct 29, 2020

Choose a reason for hiding this comment

pritamdamania87 Oct 29, 2020

Choose a reason for hiding this comment

pritamdamania87 Oct 29, 2020

Choose a reason for hiding this comment

pritamdamania87 Oct 29, 2020

Choose a reason for hiding this comment

pritamdamania87 Oct 29, 2020

Choose a reason for hiding this comment

pritamdamania87 Oct 29, 2020

Choose a reason for hiding this comment

facebook-github-bot commented Nov 3, 2020

codecov bot commented Nov 3, 2020

Codecov Report

mrshenli left a comment

Choose a reason for hiding this comment

BorisValkov5 commented Nov 4, 2020

facebook-github-bot commented Nov 5, 2020

dr-ci bot commented Oct 28, 2020 •

edited