Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A bug when retraining #6

Closed
shgidi opened this issue Feb 12, 2020 · 0 comments
Closed

A bug when retraining #6

shgidi opened this issue Feb 12, 2020 · 0 comments

Comments

@shgidi
Copy link

shgidi commented Feb 12, 2020

Hi, when running python main.py --phase=train
I get the following error:

start training
max_step = 40000, max_ep = 223, num_batch = 179
Start Training!
Traceback (most recent call last):
  File "main.py", line 310, in <module>
    main(FLAGS)
  File "main.py", line 300, in main
    model.train(loader_dict, num_batch)
  File "main.py", line 144, in train
    [loss_value, update_value] = sess.run([loss, optim])	
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 877, in run
    run_metadata_ptr)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1100, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
    run_metadata)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0)
	 [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_UINT8], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]
	 [[Node: shuffle_batch/_31 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_173_shuffle_batch", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

Caused by op u'shuffle_batch', defined at:
  File "main.py", line 310, in <module>
    main(FLAGS)
  File "main.py", line 296, in main
    loader_dict, num_batch = data_loader_bd_rm_from_tfrecord(batch_size=1)
  File "/home/jupyter/cloned/DeepFloorplan/net.py", line 25, in data_loader_bd_rm_from_tfrecord
    loader_dict = read_bd_rm_record('../dataset/r3d.tfrecords', batch_size=batch_size, size=512)
  File "./utils/tf_record.py", line 354, in read_bd_rm_record
    batch_size=batch_size, capacity=batch_size*256, num_threads=1, min_after_dequeue=batch_size*32)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 1300, in shuffle_batch
    name=name)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 846, in _shuffle_batch
    dequeued = queue.dequeue_many(batch_size, name=name)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 476, in dequeue_many
    self._queue_ref, n=n, component_types=self._dtypes, name=name)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 3480, in queue_dequeue_many_v2
    component_types=component_types, timeout_ms=timeout_ms, name=name)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
    return func(*args, **kwargs)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
    op_def=op_def)
  File "/home/jupyter/miniconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1717, in __init__
    self._traceback = tf_stack.extract_stack()

OutOfRangeError (see above for traceback): RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 1, current size 0)
	 [[Node: shuffle_batch = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_UINT8], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]
	 [[Node: shuffle_batch/_31 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_173_shuffle_batch", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

My installments are as defined in to README.md. What can be the problem?

@shgidi shgidi closed this as completed Feb 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant