Skip to content
Permalink
Browse files

Fixes spacing in Horovod with Tensorflow docs page (#1202)

Signed-off-by: Igor Wilbert <igor.wilbert@gmail.com>
  • Loading branch information...
IgorWilbert authored and alsrgv committed Jul 10, 2019
1 parent d902152 commit e9ea85e0c5309da146a42cb9c32a5ef8fadfec19
Showing with 23 additions and 6 deletions.
  1. +23 −6 docs/summary.rst
@@ -135,40 +135,57 @@ To use Horovod, make the following additions to your program. This example uses

1. Run ``hvd.init()``.

|
.. raw:: html

<p/>

2. Pin a server GPU to be used by this process using ``config.gpu_options.visible_device_list``.

With the typical setup of one GPU per process, you can set this to *local rank*. In that case, the first process on
the server will be allocated the first GPU, the second process will be allocated the second GPU, and so forth.

|
.. raw:: html

<p/>


3. Scale the learning rate by the number of workers.

Effective batch size in synchronous distributed training is scaled by the number of workers.
An increase in learning rate compensates for the increased batch size.

|
.. raw:: html

<p/>


4. Wrap the optimizer in ``hvd.DistributedOptimizer``.

The distributed optimizer delegates gradient computation to the original optimizer, averages gradients using *allreduce* or *allgather*, and then applies those averaged gradients.

|
.. raw:: html

<p/>


5. Add ``hvd.BroadcastGlobalVariablesHook(0)`` to broadcast initial variable states from rank 0 to all other processes.

This is necessary to ensure consistent initialization of all workers when training is started with random weights or restored from a checkpoint.
Alternatively, if you're not using ``MonitoredTrainingSession``, you can execute the ``hvd.broadcast_global_variables`` op after global variables have been initialized.

|
.. raw:: html

<p/>


6. Modify your code to save checkpoints only on worker 0 to prevent other workers from corrupting them.

Accomplish this by passing ``checkpoint_dir=None`` to ``tf.train.MonitoredTrainingSession`` if ``hvd.rank() != 0``.

|
.. raw:: html

<p/>


Example (see the `examples <https://github.com/horovod/horovod/blob/master/examples/>`_ directory for full training examples):

0 comments on commit e9ea85e

Please sign in to comment.
You can’t perform that action at this time.