PyTorch Synthetic Benchmark

[alsrgv]

No description provided.

[alsrgv]

@tgaddair, we should fix this bug before landing. optim.SGD w/o momentum & weight decay cause the following issue with broadcast_optimizer_state:

Traceback (most recent call last):
  File "pytorch_synthetic_benchmark.py", line 64, in <module>
    hvd.broadcast_optimizer_state(optimizer, root_rank=0)
  File "/usr/local/lib/python2.7/site-packages/horovod/torch/__init__.py", line 213, in broadcast_optimizer_state
    param_state = state_dict['state'][pid]
KeyError: 4592867280

[tgaddair]

Done #548.

[tgaddair]

Done #548.

[bapriddy]

@alsrgv I'm getting the following error when running the pytorch_synthetic_benchmark.py

Traceback (most recent call last): File "syn.py", line 64, in <module> hvd.broadcast_optimizer_state(optimizer, root_rank=0) File "~/miniconda3/lib/python3.6/site-packages/horovod/torch/__init__.py", line 213, in broadcast_optimizer_state param_state = state_dict['state'][pid] KeyError: 140230636548384

I get the error in parallel as well. Same output k times...For example

aprun -n 4 -N 1 ~/miniconda3/bin/python syn.py

Gives
Traceback (most recent call last): File "syn.py", line 64, in <module> Traceback (most recent call last): File "syn.py", line 64, in <module> Traceback (most recent call last): File "syn.py", line 64, in <module> Traceback (most recent call last): File "syn.py", line 64, in <module> hvd.broadcast_optimizer_state(optimizer, root_rank=0) File "~/miniconda3/lib/python3.6/site-packages/horovod/torch/__init__.py", line 213, in broadcast_optimizer_state hvd.broadcast_optimizer_state(optimizer, root_rank=0) File "~/miniconda3/lib/python3.6/site-packages/horovod/torch/__init__.py", line 213, in broadcast_optimizer_state hvd.broadcast_optimizer_state(optimizer, root_rank=0) File "~/miniconda3/lib/python3.6/site-packages/horovod/torch/__init__.py", line 213, in broadcast_optimizer_state hvd.broadcast_optimizer_state(optimizer, root_rank=0) File "~/miniconda3/lib/python3.6/site-packages/horovod/torch/__init__.py", line 213, in broadcast_optimizer_state param_state = state_dict['state'][pid] param_state = state_dict['state'][pid] KeyError: 46913509835904 KeyError: 46913509835904 param_state = state_dict['state'][pid] KeyError: 46913509835904 param_state = state_dict['state'][pid] KeyError: 46913509835904

I've included the reference call below in /torch/init.py

def _create_callback(pid, name, t, p):
def _from_tensor():
state_dict['state'][pid][name] = t(p.numpy()[0])
return _from_tensor

[alsrgv]

@bapriddy, can you comment out this line or install Horovod from master?

[bapriddy]

Yes. I'll check it. Thanks.

[bapriddy]

It worked. Thanks!!

[bapriddy]

@alsrgv Awesome!!! So nice to have this. Again, Thanks!!!

[bapriddy]

@alsrgv How does the code decide when to stop "warmup" and proceed with the test?? Just curious. Also, would switching to fp16 have any effect?

[alsrgv]

@bapriddy, warmup runs for --num-warmup-batches, which defaults to 10. Using --fp16-allreduce should improve performance if your network is slow.

[bapriddy]

@alsrgv Is it possible to modify the pytorch_synthetic_benchmark.py for resnet18, resnet101, or other imagenet models? I did this with pytorch_imagenet_resnet50.py by changing line 114.

# Set up standard ResNet-50 model.
model = models.resnet50()

[alsrgv]

@bapriddy, yeah, you can just pass --model resnet101.

[bapriddy]

@alsrgv Got it!