"UnavailableError: OS Error" when running training on Google Cloud with TensorFlow 1.8

[ProjectDent]

My model trains fine in 20 minutes with TensorFlow 1.2. I changed my cloud.yml file's runtimeVersion to TensorFlow 1.8, my setup.py file's REQUIRED PACKAGES to require 'Tensorflow>=1.8.0', and my submit training command's runtime-version to 1.8.

Now, training took about 80 minutes, before crashing at 4940 steps (60 short of my 5000 steps) I'd set in my training, with this error:

The replica worker 2 exited with a non-zero status of 1. Termination reason: Error. Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/root/.local/lib/python2.7/site-packages/object_detection/train.py", line 184, in tf.app.run() File "/root/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 126, in run _sys.exit(main(argv)) File "/root/.local/lib/python2.7/site-packages/object_detection/train.py", line 180, in main graph_hook_fn=graph_rewriter_fn) File "/root/.local/lib/python2.7/site-packages/object_detection/trainer.py", line 399, in train saver=saver) File "/root/.local/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 769, in train sess, train_op, global_step, train_step_kwargs) File "/root/.local/lib/python2.7/site-packages/tensorflow/contrib/slim/python/slim/learning.py", line 487, in train_step run_metadata=run_metadata) File "/root/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 900, in run run_metadata_ptr) File "/root/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1135, in _run feed_dict_tensor, options, run_metadata) File "/root/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run run_metadata) File "/root/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call raise type(e)(node_def, op, message) UnavailableError: OS Error

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
What is the top-level directory of the model you are using
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

[wpp]

Hi, I'm experiencing the same behaviour (and error). After training successfully for a while (see below for details). The master-replica-0 spits out:

Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.UnavailableError'>, OS Error [[Node: clip_grads/clip_by_norm_2/truediv_S5417 = _Recvclient_terminated=false, recv_device="/job:ps/replica:0/task:2/device:CPU:0", send_device="/job:master/replica:0/task:0/device:CPU:0", send_device_incarnation=2772952984520217269, tensor_name="edge_8753_clip_grads/clip_by_norm_2/truediv", tensor_type=DT_FLOAT, _device="/job:ps/replica:0/task:2/device:CPU:0"]] [[Node: Momentum/update/NoOp_2_S5432 = _Recvclient_terminated=false, recv_device="/job:master/replica:0/task:0/device:CPU:0", send_device="/job:ps/replica:0/task:2/device:CPU:0", send_device_incarnation=3296691442993841585, tensor_name="edge_8991_Momentum/update/NoOp_2", tensor_type=DT_FLOAT, _device="/job:master/replica:0/task:0/device:CPU:0"]] [[Node: clip_grads/clip_by_norm_186/truediv_S5067 = _Recvclient_terminated=false, recv_device="/job:ps/replica:0/task:1/device:CPU:0", send_device="/job:master/replica:0/task:0/device:CPU:0", send_device_incarnation=2772952984520217269, tensor_name="edge_4785_clip_grads/clip_by_norm_186/truediv", tensor_type=DT_FLOAT, _device="/job:ps/replica:0/task:1/device:CPU:0"]]

Which is followed by a tear-down of the job.

This happened once after ~40mins and once after ~2hours. It was suggested elsewhere that this might be a OOM error, but master as well as workers hover at around 20% mem usage.

Let me know if and how I can provide more information.
Regards

[gamcoh]

I have the same issue when i train the faster_rcnn_resnet101 model with ml-engine

It's my trial account i have a tesla k80

here's my cloud.yml config file:

trainingInput:
  runtimeVersion: "1.9"
  scaleTier: CUSTOM
  masterType: standard_gpu
  workerCount: 3
  workerType: standard_gpu
  parameterServerCount: 3
  parameterServerType: standard

[prameshbajra]

Did anyone solve this? If yes, mind sharing how?

[prameshbajra]

@gamcoh did you solve it?

[devAdvc]

@prameshbajra nop I'm waiting too

[prameshbajra]

@devAdvc have you tried reducing the number of workerCount: in your cloud.yml file?

[moussas1]

Anyone solved this issue?

[prameshbajra]

@moussas1 I'm stuck too. Let me know if you happen to solve it

[Paulik8]

As i understand this is bug Tensorflow developers know about but don't fix it!(i don't undestand why)
I tried to change this (https://github.com/tensorflow/tensorflow/blob/09ec6b2c59d05230b275a14bbc54a8890b6b1c62/tensorflow/contrib/slim/python/slim/learning.py#L721) as described below but I CANT edit system files in google cloudshell...
#3788 (comment)

P S learning.py i could find thanks to tensorflow/tensorflow#15793 (comment)

Like solution u can reduce runtime version and hope that will be worked)

After few hours i find solution and test it.
Just set this environment variable on your pc before start training:
export GRPC_POLL_STRATEGY=poll
It works on tensorflow 1.10 and my cloud.yml:

 trainingInput:
  scaleTier: CUSTOM
  masterType: standard_gpu
  workerCount: 5
  workerType: standard_gpu
  parameterServerCount: 3
  parameterServerType: standard
  pythonVersion: "3.5"

[pkulzc]

This should be gone if you sync to HEAD and use tf 1.12.

[moussas1]

Ok I will update to 1.12, but what do you mean sync to HEAD? Thank you!

[pkulzc]

I mean have you synced your fork?

[Paulik8]

@pkulzc, is synced fork the same as git pull?
Btw I did git pull and all works for ssdlite_mobilinet_v2_coco but not for ssd_mobilenet_v2_quantized. The last one continue to have UnavailableError: OS error

[prameshbajra]

@Paulik8 @moussas1 @devAdvc Did any of you get past this error? It did not work for me even when I sync to head.

[moussas1]

@prameshbajra Still nothing for me. :/ #6220 I also posted a full log of things in the referenced issue. @pkulzc Any ideas? Thanks

[pkulzc]

For future viewers, please switch to another thread to track the progress of this issue since this one is duplicated.

[tensorflowbutler]

Hi There,
We are checking to see if you still need help on this, as this seems to be considerably old issue. Please update this issue with the latest information, code snippet to reproduce your issue and error you are seeing.
If we don't hear from you in the next 7 days, this issue will be closed automatically. If you don't need help on this issue any more, please consider closing this.