InvalidArgumentError: Nan in summary histogram

[LinZhineng]

I try to run through the face generator by default parameter using celebA, the project works well until it crashes, and I get the following error:

Traceback (most recent call last):
File "main.py", line 58, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "main.py", line 42, in main
dcgan.train(FLAGS)
File "/home/linzhineng/DCGAN-tensorflow/model.py", line 153, in train
feed_dict={ self.images: batch_images, self.z: batch_z })
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 372, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 636, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 708, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 728, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: Nan in summary histogram for: HistogramSummary_1
[[Node: HistogramSummary_1 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](HistogramSummary_1/tag, Sigmoid/_39)]]
Caused by op u'HistogramSummary_1', defined at:
File "main.py", line 58, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv))
File "main.py", line 39, in main
dataset_name=FLAGS.dataset, is_crop=FLAGS.is_crop, checkpoint_dir=FLAGS.checkpoint_dir)
File "/home/linzhineng/DCGAN-tensorflow/model.py", line 64, in init self.build_model()
File "/home/linzhineng/DCGAN-tensorflow/model.py", line 85, in build_model
self.d_sum = tf.histogram_summary("d", self.D)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/logging_ops.py", line 125, in histogram_summary
tag=tag, values=values, name=scope)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 100, in _histogram_summary
name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/op_def_library.py", line 704, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2260, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1230, in init
self._traceback = _extract_stack()

It occurs in the first epoch. the same error occurs in the project dcgan.torch

any advice? I find that d_loss may suddenly fall fast, the possible reasons is D_network still convergence fast by default parameter.

[minimumnz]

I had this issue with TensorFlow 0.8 upgrading fixed it.

[myw8]

Hi, I have encountered the same problem, can you tell me how you solve the problem? And I tried 0.8,0.9 . Still the same problem

[maz369]

I am facing the same problem (python 2.7; tf 0.12.1). Would appreciate it if you could let us know how you solved it? Which version of tensorflow are you using?

Thank you

[aocz1]

I've been having the same issue, it turns out using version tensorflow==1.0 fixes this