I meet the same problem @skye in using object detection apis

Can you provide code to repro the problem?

Rohan/Priya: I'm guessing this is what happens when a tf.string tensor goes through prefetch_to_devices(), but unclear whether it should be handled in the client program (e.g. by splitting out strings from the prefetched dataset) or in the FunctionBufferingResource (e.g. by allowing some outputs to be "host memory" only).

def get_inputs(mode, csv_file, batch_size, label_list, preprocess):
iterator_initializer_hook = IteratorInitializerHook()
def inputs():
is_training = mode==estimator.ModeKeys.TRAIN
ds = tf.data.TextLineDataset(csv_file).skip(1)

    def classification_parse_line(line):
        columns = ['img','label']
        img_name, label = tf.decode_csv(
            line, 
            record_defaults = [[''],['']]) 
        # assume every pic is rgb
        image_decoded = tf.image.decode_png(
            tf.read_file(img_name),
            channels=3)
        image = preprocess(image_decoded)
        """image = image_preprocessing_fn(
            image, 
            image_size,
            image_size)"""
        return image,label
    
    cpu_num = multiprocessing.cpu_count()
    ds = ds.apply(
        tf.contrib.data.map_and_batch(
            classification_parse_line,
            batch_size=batch_size,
            num_parallel_batches=cpu_num))   
    ds = ds.prefetch(None)
    iterator = ds.make_initializable_iterator()

    iterator_initializer_hook.iterator_initializer_func = lambda sess: sess.run(iterator.initializer)
    return ds

return iterator_initializer_hook, inputs

distribution = tf.contrib.distribute.MirroredStrategy()
config = tf.estimator.RunConfig(
model_dir=args.model_dir,
tf_random_seed=912,
save_summary_steps=args.save_summary_steps,
save_checkpoints_steps=args.save_interval_steps,
keep_checkpoint_max=5*get_num_replicas(),
train_distribute=distribution,
session_config=session_config
)

classifier = tf.estimator.Estimator(
my_model,
config=config,
params=params)
for epoch in range(args.num_epochs):
logger.info('Starting epoch %d / %d' % (
epoch + 1, args.num_epochs))
classifier.train(
train_ds,
hooks=[train_ds_hook])
classifier.evaluate(
val_ds,
hooks=[val_ds_hook])

Nothing special, its just common code with MirroredStrategy.
If replace MirroredStrategy with OneDeviceStrategy, everything went well

Nagging Assignees @rohan100jain, @guptapriya: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

As Derek mentioned, currently we don't have a mechanism for specifying if some outputs should be in host memory and we assume (to a large extent) that they'd be in device memory. Strings can't be in device memory, hence the bug. I shall work on having a dynamic method of identifying which outputs should be allocated on the host / device. Stay tuned for a fix in a bit.

@rohan100jain can this issue be closed now that your fix has been merged?