A very weird bug of tf_debug: Non-Ok-status env->NewWritableFile(file_path, &f) status: Resource exaustated

[ghost]

System information

• Have I written custom code: Yes
• OS Platform and Distribution: CentOS
• TensorFlow installed from (source or binary): binary
• TensorFlow version: 1.2
• Python version: 3.4

When I run the seq2seq model, I got the nan loss. Thus I use tf_debug to find out where the problem occurs. I use tf_debug by

sv = tf.train.Supervisor(logdir=FLAGS.log_root,
                         is_chief=True,
                         saver=saver,
                         summary_op=None,
                         save_summaries_secs=60,
                         save_model_secs=FLAGS.checkpoint_secs)
sess = sv.prepare_or_wait_for_session(config=tf.ConfigProto(
    allow_soft_placement=True))
sess = tf_debug.LocalCLIDebugWrapperSession(sess)
sess.add_tensor_filter("has_inf_or_nan", tf_debug.has_inf_or_nan)

But it got such logs and exit:

tfdebug Non-OK-status: env->NewWritableFile(file_path, &f) status: Resource exaustated: /tmp/tfdbg_9_gcc3sc/gradients/output/Reshape_1_grad/Reshape_0_DebugIdentity_150023213
Aborted (core dumped)

I think it is kind of issue related to tf_debug and it may be new, since I can not find anything when I google the error.

[asimshankar]

This error occurs when writing to the filesystem fails - for example if there is no space left on the device, or if the disk quota exceeded or if there were too many files etc. (see errors.cc)

Unfortunately, the error message given doesn't give more detail (I'll look into fixing that), but the underlying problem does appear to be that you're hitting some sort of space or user limit issues when writing to /tmp/tfdbg_9_gcc3sc/gradients/output/Reshape_1_grad/Reshape_0_DebugIdentity_150023213

Could you look into how much space is left on the device and whether you're running into disk limits?

[ghost]

@asimshankar Thanks for your reply! But it is unlikely that no space left on the device. Cause I run the code on a server, whose space is much more than enough. Do you think user authentication can be a possible reason. But I have no problem on writing under my path.

[asimshankar]

I didn't quite follow what you meant by "I have no problem on writing under my path". Could you elaborate on that?

But regardless, it seems like the fopen call here that is failing, so it does seem to be the filesystem rejecting the open call. So perhaps there are some ulimits in place?

[tensorflowbutler]

This issue is automatically closed due to lack of activity. Please re-open if this is still an issue for you. Thanks!