Tensorflow stucks in evaluation step after running local_init_op

[abdullahakmal]

OS: Ubuntu 18.0.41 LTS and 16.04
Tried on CPU and GPU both. Same error.
Tensorflow version 1.1, 1.5 and1.8 also.
Python 3.5 and 3.6.
Tensorflow was installed using the pip.

Command Prompt Snippet of the problem

INFO:tensorflow:Creating AttentionDecoder in mode=eval
INFO:tensorflow:
AttentionDecoder:
init_scale: 0.04
max_decode_length: 100
rnn_cell:
cell_class: LSTMCell
cell_params: {num_units: 512}
dropout_input_keep_prob: 0.8
dropout_output_keep_prob: 1.0
num_layers: 4
residual_combiner: add
residual_connections: false
residual_dense: false

INFO:tensorflow:Creating ZeroBridge in mode=eval
INFO:tensorflow:
ZeroBridge: {}

WARNING:tensorflow:From /home/muhammad/Thesis/Ab/AutoViz/seq2seq/metrics/metric_specs.py:232: streaming_mean (from tensorflow.contrib.metrics.python.ops.metric_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.metrics.mean
INFO:tensorflow:Starting evaluation at 2018-10-23-00:06:12
INFO:tensorflow:Graph was finalized.
INFO:tensorflow:Restoring parameters from /home/muhammad/Thesis/Ab/AutoViz/Model/model.ckpt-1406
INFO:tensorflow:Running local_init_op.
INFO:tensorflow:Done running local_init_op.

---

I have left it running for more than a day but it remains stuck here. On GPU it shows that the process is running, but system doesn't seems like doing anything because of the processor usage.

Any help would be appreaciated a lot.

Thank You.

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
What is the top-level directory of the model you are using
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

[abdullahakmal]

What is the top-level directory of the model you are using: N/A
Have I written custom code: N/A
OS Platform and Distribution: Windows 10 and Ubuntu (18.04 + 16.04)
TensorFlow installed from: Python-Pip (TensorFlow Versions used: 1.0, 1.1 and 1.8)
Bazel version: N/A
CUDA/cuDNN version: 8 and 9
GPU model and memory: 940M (2GB)
Exact command to reproduce: Just running the training sequence in Python 3.5 and Python 3.6.

[BowieHsu]

@abdullahakmal does checkpoint files produced on the destination folder? maybe it's just because the tensorflow logging function is turned off in default setting.

[abdullahakmal]

No it produces checkpoints successfully.

[BowieHsu]

@abdullahakmal yeah, i mean the program is running, just turned off the logging info, you may write "tf.logging.set_verbosity(tf.logging.INFO)" in model_main.py, so the program will logging the loss on terminal

[abdullahakmal]

Okay. I'll try that.

[abdullahakmal]

tf.logging.set_verbosity(tf.logging.INFO)
is already set.

[beerys]

Was this resolved? I've just seen similar behavior on one of my jobs.

[KuribohG]

The same happens to me.
I am training an object detector using a custom dataset. What is your task?
Have anyone resolved this?

[alk15]

Any updates on that?
Execution also hangs after I get the following message:
INFO:tensorflow:Done running local_init_op.
I am able to continue the training normally by using Ctrl+C and re-starting from latest checkpoint.

[abdullahakmal]

I moved to torch....

…

On Wed, 6 Feb 2019 at 21:39, alk15 ***@***.***> wrote: Any updates on that? Execution also hangs after I get the following message: INFO:tensorflow:Done running local_init_op. I am able to continue the training normally by using Ctrl+C and re-starting from latest checkpoint. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5587 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AY7cZS3AD2r_13_Tcv373IDOKls4PUFDks5vK0t5gaJpZM4X2X33> .

-- Best Regards, *Muhammad Abdullah Akmal* Data Scientist at Gamesessions, Sheffield, South Yorkshire, United Kingdom

[LeoHirasawa]

I got the same issue, and I checked my code,I found there is a line:
dataset = dataset.repeat()
in my eval_input_fn() function.
I removed it,and the issue gone.
THIS LINE OF CODE WILL LEAD YOUR EVALUATE PROGRESS INTO A DEAD CYCLE