OutOfRangeError (see above for traceback): FIFOQueue '_3_batch/fifo_queue' is closed and has insufficient elements (requested 16, current size 0)

[kentrabbitwjl]

Hi,

I get the following error when attempting to train the Fg Bg reconstruction model, using tensorflow-gpu1.9, python2.7. Thank you for help.

gpu=0
D_arch='DCGAN'
log_dir='./logs'

####################### Stage-I: reconstruction #####################
## Fg Bg reconstruction
model_dir=${log_dir}'/MODEL1_Encoder_GAN_BodyROI7'
python main.py --dataset=Market_train_data \
             --use_gpu=True --img_H=128  --img_W=64 \
             --batch_size=16 --max_step=120000 \
             --d_lr=0.00002  --g_lr=0.00002 \
             --lr_update_step=50000 \
             --model=1 \
             --D_arch=${D_arch} \
             --gpu=${gpu} \
             --z_num=64 \
             --model_dir=${model_dir} \

error message:
/home/ryu/sourcecode/Disentangled-Person-Image-Generation/tflib/plot.py:4: UserWarning:
This call to matplotlib.use() has no effect because the backend has already
been chosen; matplotlib.use() must be called before pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

The backend was originally set to 'TkAgg' by the following code:
File "main.py", line 4, in
from trainer import *
File "/home/ryu/sourcecode/Disentangled-Person-Image-Generation/trainer.py", line 20, in
import models
File "/home/ryu/sourcecode/Disentangled-Person-Image-Generation/models.py", line 6, in
from utils import *
File "/home/ryu/sourcecode/Disentangled-Person-Image-Generation/utils.py", line 492, in
import matplotlib.pyplot as plt
File "/home/ryu/anaconda3/envs/pose/lib/python2.7/site-packages/matplotlib/pyplot.py", line 71, in
from matplotlib.backends import pylab_setup
File "/home/ryu/anaconda3/envs/pose/lib/python2.7/site-packages/matplotlib/backends/init.py", line 16, in
line for line in traceback.format_stack()

matplotlib.use('Agg')
load pn_pairs_num ......
######coord2channel_simple#####
######coord2channel_simple#####
Uppercase local vars:
BATCH_SIZE: 16
DATA_DIR:
DIM: 64
G_OUTPUT_DIM: 24576
IMG_H: 128
IMG_W: 64
ITERS: 200000
LAMBDA: 10
MODE: dcgan
WARNING:tensorflow:From /home/ryu/sourcecode/Disentangled-Person-Image-Generation/trainer.py:199: init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2018-07-13 10:59:03,319:WARNING::From /home/ryu/sourcecode/Disentangled-Person-Image-Generation/trainer.py:199: init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2018-07-13 10:59:06.651736: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.6325
pciBusID: 0000:02:00.0
totalMemory: 10.92GiB freeMemory: 10.76GiB
2018-07-13 10:59:06.651792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2018-07-13 10:59:07.038518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-13 10:59:07.038577: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958] 0
2018-07-13 10:59:07.038589: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0: N
2018-07-13 10:59:07.038950: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10411 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1)
[] MODEL dir: ./logs/MODEL1_Encoder_GAN_BodyROI7
[] PARAM path: ./logs/MODEL1_Encoder_GAN_BodyROI7/params.json
2018-07-13 10:59:21.620532: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: part_bbox_1. Can't parse serialized Example.
2018-07-13 10:59:21.620573: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: part_bbox_1. Can't parse serialized Example.
2018-07-13 10:59:21.621055: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: part_bbox_1. Can't parse serialized Example.
2018-07-13 10:59:21.621579: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at example_parsing_ops.cc:240 : Invalid argument: Key: part_bbox_1. Can't parse serialized Example.
Traceback (most recent call last):
File "main.py", line 90, in
main(config)
File "main.py", line 82, in main
trainer.train()
File "/home/ryu/sourcecode/Disentangled-Person-Image-Generation/trainer.py", line 328, in train
part_bbox_target_fixed, part_vis_fixed, part_vis_target_fixed = self.get_image_from_loader()
File "/home/ryu/sourcecode/Disentangled-Person-Image-Generation/trainer.py", line 530, in get_image_from_loader
self.mask_r6, self.mask_r6_target, self.part_bbox, self.part_bbox_target, self.part_vis, self.part_vis_target])
File "/home/ryu/anaconda3/envs/pose/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/ryu/anaconda3/envs/pose/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ryu/anaconda3/envs/pose/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/ryu/anaconda3/envs/pose/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: FIFOQueue '_3_batch/fifo_queue' is closed and has insufficient elements (requested 16, current size 0)
[[Node: batch = QueueDequeueManyV2[component_types=[DT_UINT8, DT_UINT8, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64, DT_INT64, DT_INT64, DT_INT64], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](batch/fifo_queue, batch/n)]]

Caused by op u'batch', defined at:
File "main.py", line 90, in
main(config)
File "main.py", line 23, in main
trainer = DPIG_Encoder_GAN_BodyROI_FgBg(config)
File "/home/ryu/sourcecode/Disentangled-Person-Image-Generation/trainer.py", line 42, in init
self.part_bbox, self.part_bbox_target, self.part_vis, self.part_vis_target = self._load_batch_pair_pose(self.dataset_obj)
File "/home/ryu/sourcecode/Disentangled-Person-Image-Generation/trainer.py", line 555, in _load_batch_pair_pose
batch_size=self.batch_size, num_threads=self.num_threads, capacity=self.capacityCoff * self.batch_size)
File "/home/ryu/anaconda3/envs/pose/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 988, in batch
name=name)
File "/home/ryu/anaconda3/envs/pose/lib/python2.7/site-packages/tensorflow/python/training/input.py", line 762, in _batch
dequeued = queue.dequeue_many(batch_size, name=name)
File "/home/ryu/anaconda3/envs/pose/lib/python2.7/site-packages/tensorflow/python/ops/data_flow_ops.py", line 483, in dequeue_many
self._queue_ref, n=n, component_types=self._dtypes, name=name)
File "/home/ryu/anaconda3/envs/pose/lib/python2.7/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 3480, in queue_dequeue_many_v2
component_types=component_types, timeout_ms=timeout_ms, name=name)
File "/home/ryu/anaconda3/envs/pose/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/ryu/anaconda3/envs/pose/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/home/ryu/anaconda3/envs/pose/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1740, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

OutOfRangeError (see above for traceback): FIFOQueue '_3_batch/fifo_queue' is closed and has insufficient elements (requested 16, current size 0)
[[Node: batch = QueueDequeueManyV2[component_types=[DT_UINT8, DT_UINT8, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_INT64, DT_INT64, DT_INT64, DT_INT64], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](batch/fifo_queue, batch/n)]]