Pop from empty context_switches when take outputs of one estimator.predict as inputs of another #20506

miacro · 2018-07-03T09:12:58Z

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): y
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux 4.14.8-gentoo-r1
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 1.8.0
Python version: 3.5.5
Bazel version (if compiling from source): n
GCC/Compiler version (if compiling from source): n
CUDA/cuDNN version: 7.1.4
GPU model and memory: GeForce GTX 1080 Ti / 11171MiB
Exact command to reproduce:

Describe the problem

Currently I am working on making predictions of two models (outputs of model1 as inputs of model2) with large amount of inputs. I tried to use tf.data.Dataset.from_generator, while it seems there are some problems about context stack.

Source code / logs

Here is my example

import tensorflow as tf

def model_fn(features, labels, mode):
    x = features["inputs"]
    W = tf.get_variable(
        name="weight",
        shape=[1],
        dtype=tf.float32,
        initializer=tf.initializers.constant(value=0.0),
        trainable=(mode == tf.estimator.ModeKeys.TRAIN),
    )
    b = tf.get_variable(
        name="bias",
        shape=[1],
        dtype=tf.float32,
        initializer=tf.initializers.constant(value=0.0),
        trainable=(mode == tf.estimator.ModeKeys.TRAIN),
    )
    hypothesis = W * x + b
    if mode == tf.estimator.ModeKeys.TRAIN:
        y = labels["labels"]
        cost = tf.square(hypothesis - y)
        optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01)
        train_op = optimizer.minimize(
            loss=cost, global_step=tf.train.get_global_step())
    else:
        train_op = None
        cost = None
    return tf.estimator.EstimatorSpec(
        mode=mode, predictions=hypothesis, loss=cost, train_op=train_op)


def get_input_fn(inputs, num_epochs=1):
    def gen():
        for item in inputs:
            if isinstance(item, tuple) or isinstance(item, list):
                yield item
            else:
                yield item, 0

    def input_fn():
        dataset = tf.data.Dataset.from_generator(
            gen, (tf.float32, tf.float32),
            (tf.TensorShape([]), tf.TensorShape([])))
        dataset = dataset.repeat(num_epochs)
        dataset = dataset.batch(1)
        iterator = dataset.make_one_shot_iterator()
        inputs, labels = iterator.get_next()
        return {"inputs": inputs}, {"labels": labels}

    return input_fn


model1 = tf.estimator.Estimator(model_fn=model_fn, model_dir="model1")
model2 = tf.estimator.Estimator(model_fn=model_fn, model_dir="model2")
# model1.train(input_fn=get_input_fn([(1, 1), (2, 2), (3, 3)], num_epochs=100))
# model2.train(input_fn=get_input_fn([(1, 2), (2, 4), (3, 6)], num_epochs=100))

# ok
for item in model1.predict(input_fn=get_input_fn([1, 2, 3, 4])):
    print(item)
# ok
for item in model2.predict(input_fn=get_input_fn([1, 2, 3, 4])):
    print(item)

model1_output = model1.predict(input_fn=get_input_fn([1, 2, 3, 4]))
model2_input_fn = get_input_fn(model1_output)
# IndexError: pop from empty list
# tensorflow/python/framework/ops.py", line 5267, in get_controller
#    context.context().context_switches.pop()
for item in model2.predict(model2_input_fn):
    print(item)

Error log

Traceback (most recent call last):

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 157, in __call__
    ret = func(*args)

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 382, in generator_py_func
    values = next(generator_state.get_iterator(iterator_id))

  File "test.py", line 36, in gen
    for item in inputs:

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 519, in predict
    for key, value in six.iteritems(preds_evaluated)

  File "/usr/lib64/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/framework/ops.py", line 5267, in get_controller
    context.context().context_switches.pop()

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/eager/context.py", line 136, in pop
    self.stack.pop()

IndexError: pop from empty list


Traceback (most recent call last):
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: IndexError: pop from empty list
Traceback (most recent call last):

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 157, in __call__
    ret = func(*args)

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 382, in generator_py_func
    values = next(generator_state.get_iterator(iterator_id))

  File "test.py", line 36, in gen
    for item in inputs:

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 519, in predict
    for key, value in six.iteritems(preds_evaluated)

  File "/usr/lib64/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/framework/ops.py", line 5267, in get_controller
    context.context().context_switches.pop()

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/eager/context.py", line 136, in pop
    self.stack.pop()

IndexError: pop from empty list


         [[Node: PyFunc = PyFunc[Tin=[DT_INT64], Tout=[DT_FLOAT, DT_FLOAT], token="pyfunc_7"](arg0)]]
         [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[?], [?]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](OneShotIterator)]]
         [[Node: IteratorGetNext/_19 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_6_IteratorGetNext", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 72, in <module>
    for item in model2.predict(model2_input_fn):
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 509, in predict
    preds_evaluated = mon_sess.run(predictions)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 567, in run
    run_metadata=run_metadata)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1043, in run
    run_metadata=run_metadata)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1134, in run
    raise six.reraise(*original_exc_info)
  File "/usr/lib64/python3.5/site-packages/six.py", line 693, in reraise
    raise value
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1119, in run
    return self._sess.run(*args, **kwargs)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1191, in run
    run_metadata=run_metadata)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 971, in run
    return self._sess.run(*args, **kwargs)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: IndexError: pop from empty list
Traceback (most recent call last):

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 157, in __call__
    ret = func(*args)

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 382, in generator_py_func
    values = next(generator_state.get_iterator(iterator_id))

  File "test.py", line 36, in gen
    for item in inputs:

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 519, in predict
    for key, value in six.iteritems(preds_evaluated)

  File "/usr/lib64/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/framework/ops.py", line 5267, in get_controller
    context.context().context_switches.pop()

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/eager/context.py", line 136, in pop
    self.stack.pop()

IndexError: pop from empty list


         [[Node: PyFunc = PyFunc[Tin=[DT_INT64], Tout=[DT_FLOAT, DT_FLOAT], token="pyfunc_7"](arg0)]]
         [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[?], [?]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](OneShotIterator)]]
         [[Node: IteratorGetNext/_19 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_6_IteratorGetNext", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

The text was updated successfully, but these errors were encountered:

aselle · 2018-07-14T00:06:09Z

@mrry, could you please take a look? (Are you the right person)?

mrry · 2018-07-15T21:24:19Z

Thanks for the straightforward reproducer, @miacro. It looks like this might be a bug in the Estimator.predict() implementation (or at least something that deserves a better error message). In particular, if I changed the following lines of code:

model1_output = model1.predict(input_fn=get_input_fn([1, 2, 3, 4]))
model2_input_fn = get_input_fn(model1_output)
for item in model2.predict(model2_input_fn):
    print(item)

...to the following:

model1_output = model1.predict(input_fn=get_input_fn([1, 2, 3, 4]))
model1_output = list(model1_output)
model2_input_fn = get_input_fn(model1_output)
for item in model2.predict(model2_input_fn):
    print(item)

...the program behaves correctly.

I'll assign this to @ispirmustafa, since he made some changes to context handling in Estimator.predict() in 0385bfe, and this may account for the error message about "context switches".

tensorflowbutler · 2018-08-31T08:08:19Z

Nagging Assignee @ispirmustafa: It has been 46 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

tensorflowbutler · 2018-09-15T18:46:48Z

Nagging Assignee @ispirmustafa: It has been 61 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

ispirmustafa · 2018-09-17T23:56:10Z

Estimator.predict yields predictions under a graph context. The way with handle graph context in TF make yielding an issue. For example following code has same error.

I'll update the documentation of predict:

def f1():
  with tf.Graph().as_default():
    yield tf.no_op()

def f2():
  with tf.Graph().as_default():
    yield tf.no_op()

y1, y2 = f1(), f2()

next(y1)
next(y2)
next(y1)

ispirmustafa · 2018-09-24T22:19:08Z

Updated the documentation: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/estimator/estimator.py#L484

3NFBAGDU · 2019-07-25T08:52:57Z

Hi, estimator.predict is to slow , i want to predict some text in my model for every 2 second. estimator.predict() function allways load the model from latest checkpoint. I want to load model only once in estimator.predict to get fast prediction.

tensorflowbutler assigned aselle Jul 3, 2018

aselle assigned mrry and unassigned aselle Jul 14, 2018

mrry assigned ispirmustafa and unassigned mrry Jul 15, 2018

ispirmustafa closed this as completed Sep 24, 2018

Harshini-Gadige added comp:apis Highlevel API related issues type:docs-bug Document issues labels Jan 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pop from empty context_switches when take outputs of one estimator.predict as inputs of another #20506

Pop from empty context_switches when take outputs of one estimator.predict as inputs of another #20506

miacro commented Jul 3, 2018

aselle commented Jul 14, 2018

mrry commented Jul 15, 2018

tensorflowbutler commented Aug 31, 2018

tensorflowbutler commented Sep 15, 2018

ispirmustafa commented Sep 17, 2018

ispirmustafa commented Sep 24, 2018

3NFBAGDU commented Jul 25, 2019

Pop from empty context_switches when take outputs of one estimator.predict as inputs of another #20506

Pop from empty context_switches when take outputs of one estimator.predict as inputs of another #20506

Comments

miacro commented Jul 3, 2018

System information

Describe the problem

Source code / logs

aselle commented Jul 14, 2018

mrry commented Jul 15, 2018

tensorflowbutler commented Aug 31, 2018

tensorflowbutler commented Sep 15, 2018

ispirmustafa commented Sep 17, 2018

ispirmustafa commented Sep 24, 2018

3NFBAGDU commented Jul 25, 2019