Error while running distributed inception tensorflow on 2 machines

[deepali-c]

Please let us know which model this issue is about (specify the top-level directory)

inception

I executed inception on a distributed TF cluster with 1 ps and 2 workers on localhost.
Next I am trying to run the sample on a distributed TF cluster with ps on one machine and 2 workers on another machine. In this case I am getting errors.

The following error is seen on worker 0:
tensorflow.python.framework.errors.NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for /tmp/imagenet_train/model.ckpt-1362

The errors on worker 1 are:

W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for /tmp/imagenet_train/model.ckpt-1362
W tensorflow/core/framework/op_kernel.cc:975] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for /tmp/imagenet_train/model.ckpt-1362

[minbinL]

worker0 saved checkpoint in machine A : /tmp/imagenet_train/
ps tried to read a latest checkpoint from /tmp/imagenet_train in machine B.
The checkpoint directory can not be reached by ps, so you need to copy /tmp/imagenet_train from A machine to B machine.

You need to make sure that the checkpoint is in a location that can be reached by all of the ps tasks

[deepali-c]

I could finally execute inception with PS on one machine and Worker on another machine. This could work even without a common shared location. However, worker 0 showed following error at the end: RuntimeError: ('Coordinator stopped with threads still running: %s', 'Thread-16').

I observed later that if I execute inception training when the /tmp/imagenet_train/ directory exists and contains checkpoints and parameters from previous run then I get the error:
tensorflow.python.framework.errors.InvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /tmp/imagenet_train/model.ckpt-88: Not found: /tmp/imagenet_train

[michaelisard]

It sounds as if you are still having trouble making the /tmp/imagenet_train directory visible to all the processes that need to see it. This kind of usage question is better asked on Stackoverflow.