How can I start a benchmark with `distributed_all_reduce` ?

[sleepfin]

My Env:
TensorFlow: 1.3
CUDA: 8.0
cuDNN: 6.0

I notice an update for distributed_all_reduce so I want to have a try. But I'm not sure what value should controller_host takes...
My args are:

--variable_update=distributed_all_reduce
--all_reduce_spec=pscpu:32k:xring

and I start 3 processes with args:
FIRST:

--job_name=worker
--worker_hosts=127.0.0.1:50001,127.0.0.1:50002
--task_index=0

SECONDE:

--job_name=worker
--worker_hosts=127.0.0.1:50001,127.0.0.1:50002
--task_index=1

THIRD:

--job_name=controller
--controller_host=??
--task_index=0

When I put 127.0.0.1:50000 or 127.0.0.1:50001 on controller_host, I got:

TensorFlow:  1.3
Model:       resnet50
Mode:        training
SingleSess:  True
Batch size:  128 global
             64 per device
Devices:     ['job:worker/task0/gpu:0', 'job:worker/task1/gpu:0']
Data format: NCHW
Optimizer:   sgd
Variables:   distributed_all_reduce
AllReduce:   pscpu:32k:xring
Sync:        True
==========
Generating model
WARNING:tensorflow:From /home/zzy/workspace/benchmarks/scripts/tf_cnn_benchmarks/preprocessing.py:486: __init__ (from tensorflow.contrib.data.python.ops.readers) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.TFRecordDataset`.
WARNING:tensorflow:From /home/zzy/workspace/benchmarks/scripts/tf_cnn_benchmarks/preprocessing.py:487: range (from tensorflow.contrib.data.python.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.range()`.
WARNING:tensorflow:From /home/zzy/workspace/benchmarks/scripts/tf_cnn_benchmarks/preprocessing.py:489: zip (from tensorflow.contrib.data.python.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.zip()`.
2017-10-10 14:03:34.183287: E tensorflow/core/common_runtime/session.cc:69] Not found: No session factory registered for the given session options: {target: "127.0.0.1:50001" config: intra_op_parallelism_threads: 1 gpu_options { force_gpu_compatible: true } allow_soft_placement: true} Registered factories are {DIRECT_SESSION, GRPC_SESSION}.
Traceback (most recent call last):
  File "/home/zzy/workspace/benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py", line 46, in <module>
    tf.app.run()
  File "/home/zzy/anaconda2/envs/tf-1.3/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/home/zzy/workspace/benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py", line 42, in main
    bench.run()
  File "/home/zzy/workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py", line 892, in run
    return self._benchmark_cnn()
  File "/home/zzy/workspace/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py", line 1068, in _benchmark_cnn
    start_standard_services=start_standard_services) as sess:
  File "/home/zzy/anaconda2/envs/tf-1.3/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/zzy/anaconda2/envs/tf-1.3/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 964, in managed_session
    self.stop(close_summary_writer=close_summary_writer)
  File "/home/zzy/anaconda2/envs/tf-1.3/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 792, in stop
    stop_grace_period_secs=self._stop_grace_secs)
  File "/home/zzy/anaconda2/envs/tf-1.3/lib/python2.7/site-packages/tensorflow/python/training/coordinator.py", line 389, in join
    six.reraise(*self._exc_info_to_raise)
  File "/home/zzy/anaconda2/envs/tf-1.3/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 953, in managed_session
    start_standard_services=start_standard_services)
  File "/home/zzy/anaconda2/envs/tf-1.3/lib/python2.7/site-packages/tensorflow/python/training/supervisor.py", line 708, in prepare_or_wait_for_session
    init_feed_dict=self._init_feed_dict, init_fn=self._init_fn)
  File "/home/zzy/anaconda2/envs/tf-1.3/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 273, in prepare_session
    config=config)
  File "/home/zzy/anaconda2/envs/tf-1.3/lib/python2.7/site-packages/tensorflow/python/training/session_manager.py", line 178, in _restore_checkpoint
    sess = session.Session(self._target, graph=self._graph, config=config)
  File "/home/zzy/anaconda2/envs/tf-1.3/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1482, in __init__
    super(Session, self).__init__(target, graph, config=config)
  File "/home/zzy/anaconda2/envs/tf-1.3/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 622, in __init__
    self._session = tf_session.TF_NewDeprecatedSession(opts, status)
  File "/home/zzy/anaconda2/envs/tf-1.3/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: No session factory registered for the given session options: {target: "127.0.0.1:50001" config: intra_op_parallelism_threads: 1 gpu_options { force_gpu_compatible: true } allow_soft_placement: true} Registered factories are {DIRECT_SESSION, GRPC_SESSION}.

[sleepfin]

Codes HERE
master_target = self.worker_hosts[0]
I think it should be:
master_target = 'grpc://' + self.worker_hosts[0]
I'm not sure what's the string when protocal is grpc+verbs

But I meet a new problem when I start 3 processes. The controller process freezes. Here's the logs:

TensorFlow:  1.3
Model:       resnet50
Mode:        training
SingleSess:  True
Batch size:  128 global
             64 per device
Devices:     ['job:worker/task0/gpu:0', 'job:worker/task1/gpu:0']
Data format: NCHW
Optimizer:   sgd
Variables:   distributed_all_reduce
AllReduce:   xring
Sync:        True
==========
Generating model
WARNING:tensorflow:From /home/zzy/workspace/benchmarks/scripts/tf_cnn_benchmarks/preprocessing.py:486: __init__ (from tensorflow.contrib.data.python.ops.readers) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.TFRecordDataset`.
WARNING:tensorflow:From /home/zzy/workspace/benchmarks/scripts/tf_cnn_benchmarks/preprocessing.py:487: range (from tensorflow.contrib.data.python.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.range()`.
WARNING:tensorflow:From /home/zzy/workspace/benchmarks/scripts/tf_cnn_benchmarks/preprocessing.py:489: zip (from tensorflow.contrib.data.python.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.zip()`.
E1010 15:32:43.587504542   18066 ev_epoll1_linux.c:1051]     grpc epoll fd: 3

I find that controller_host is passed into self.cluster and self.cluster is never used actually. So how dose controller_host work?

[sleepfin]

@tfboyd Would you pls have a look at this issue?

[tfboyd]

I am verifying the 1.4 release right now and then moving on to the latest release of the benchmark code for distributed that you are using here.

…

On Mon, Oct 16, 2017 at 5:53 AM, Zhang Zhenyu ***@***.***> wrote: @tfboyd <https://github.com/tfboyd> Would you pls have a look at this issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#64 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AWZesk1V5soMxTaE-aXMcklgFSYwMFHmks5ss1HegaJpZM4PzcKa> .

[sleepfin]

@tfboyd Any updates?

[tfboyd]

I ran the new script for benchmarks but did not have time to play with distributed. I am moving away from running theses tests. Rather I am done running them. @reedwm might be able to help you. I thought I would have time and then I ended up deciding I am not running these tests anymore.

…

On Nov 1, 2017 1:01 AM, "Zhang Zhenyu" ***@***.***> wrote: @tfboyd <https://github.com/tfboyd> Any updates? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#64 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AWZesqjaioodFRRRuIjgixQOIWK1Z9Pcks5syCVcgaJpZM4PzcKa> .

[reedwm]

I'm fairly busy now. @poxvoculi can you take a look? I can also look into this in a week or so if you're also busy.

[poxvoculi]

It's possible we need to fix benchmark_cnn.py to look up the server config properly. During development we're testing using a different server config (not gRPC), and it looks like our automated test suite doesn't cover the case of your example.

When using distributed_all_reduce the graph is built in a different way from all of the other cases. In all of the other cases each worker instance builds its own graph(s) locally (one per each GPU), and there are no edges between the graphs, only to/from the parameter servers. In distributed_all_reduce the special controller process builds one graph spanning all devices then partitions the graph, registers each partition with the worker that hosts it then issues RunStep requests to each worker in parallel. This latter mode is denoted 'single_session' in the benchmark_cnn.py source since one session is distributed across all the workers, which are using Send/Recv graph edges among themselves to do the gradient reductions.

[atinsood]

@poxvoculi any chance this was addressed. Or anything that can be changed in the scripts locally to run these tests locally

[Luo-Liang]

Would like to know if there's any update too!

Thanks!

[reedwm]

In distributed_all_reduce, we never tf.train.Server on the workers, which I believe is required in distributed TensorFlow. This prevents us from connecting to the workers. I'll try to get this resolved.