Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pop from empty context_switches when take outputs of one estimator.predict as inputs of another #20506

Closed
miacro opened this issue Jul 3, 2018 · 6 comments

Comments

Projects
None yet
6 participants
@miacro
Copy link

commented Jul 3, 2018

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): y
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux 4.14.8-gentoo-r1
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): 1.8.0
  • Python version: 3.5.5
  • Bazel version (if compiling from source): n
  • GCC/Compiler version (if compiling from source): n
  • CUDA/cuDNN version: 7.1.4
  • GPU model and memory: GeForce GTX 1080 Ti / 11171MiB
  • Exact command to reproduce:

Describe the problem

Currently I am working on making predictions of two models (outputs of model1 as inputs of model2) with large amount of inputs. I tried to use tf.data.Dataset.from_generator, while it seems there are some problems about context stack.

Source code / logs

Here is my example

import tensorflow as tf

def model_fn(features, labels, mode):
    x = features["inputs"]
    W = tf.get_variable(
        name="weight",
        shape=[1],
        dtype=tf.float32,
        initializer=tf.initializers.constant(value=0.0),
        trainable=(mode == tf.estimator.ModeKeys.TRAIN),
    )
    b = tf.get_variable(
        name="bias",
        shape=[1],
        dtype=tf.float32,
        initializer=tf.initializers.constant(value=0.0),
        trainable=(mode == tf.estimator.ModeKeys.TRAIN),
    )
    hypothesis = W * x + b
    if mode == tf.estimator.ModeKeys.TRAIN:
        y = labels["labels"]
        cost = tf.square(hypothesis - y)
        optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01)
        train_op = optimizer.minimize(
            loss=cost, global_step=tf.train.get_global_step())
    else:
        train_op = None
        cost = None
    return tf.estimator.EstimatorSpec(
        mode=mode, predictions=hypothesis, loss=cost, train_op=train_op)


def get_input_fn(inputs, num_epochs=1):
    def gen():
        for item in inputs:
            if isinstance(item, tuple) or isinstance(item, list):
                yield item
            else:
                yield item, 0

    def input_fn():
        dataset = tf.data.Dataset.from_generator(
            gen, (tf.float32, tf.float32),
            (tf.TensorShape([]), tf.TensorShape([])))
        dataset = dataset.repeat(num_epochs)
        dataset = dataset.batch(1)
        iterator = dataset.make_one_shot_iterator()
        inputs, labels = iterator.get_next()
        return {"inputs": inputs}, {"labels": labels}

    return input_fn


model1 = tf.estimator.Estimator(model_fn=model_fn, model_dir="model1")
model2 = tf.estimator.Estimator(model_fn=model_fn, model_dir="model2")
# model1.train(input_fn=get_input_fn([(1, 1), (2, 2), (3, 3)], num_epochs=100))
# model2.train(input_fn=get_input_fn([(1, 2), (2, 4), (3, 6)], num_epochs=100))

# ok
for item in model1.predict(input_fn=get_input_fn([1, 2, 3, 4])):
    print(item)
# ok
for item in model2.predict(input_fn=get_input_fn([1, 2, 3, 4])):
    print(item)

model1_output = model1.predict(input_fn=get_input_fn([1, 2, 3, 4]))
model2_input_fn = get_input_fn(model1_output)
# IndexError: pop from empty list
# tensorflow/python/framework/ops.py", line 5267, in get_controller
#    context.context().context_switches.pop()
for item in model2.predict(model2_input_fn):
    print(item)

Error log

Traceback (most recent call last):

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 157, in __call__
    ret = func(*args)

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 382, in generator_py_func
    values = next(generator_state.get_iterator(iterator_id))

  File "test.py", line 36, in gen
    for item in inputs:

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 519, in predict
    for key, value in six.iteritems(preds_evaluated)

  File "/usr/lib64/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/framework/ops.py", line 5267, in get_controller
    context.context().context_switches.pop()

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/eager/context.py", line 136, in pop
    self.stack.pop()

IndexError: pop from empty list


Traceback (most recent call last):
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: IndexError: pop from empty list
Traceback (most recent call last):

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 157, in __call__
    ret = func(*args)

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 382, in generator_py_func
    values = next(generator_state.get_iterator(iterator_id))

  File "test.py", line 36, in gen
    for item in inputs:

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 519, in predict
    for key, value in six.iteritems(preds_evaluated)

  File "/usr/lib64/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/framework/ops.py", line 5267, in get_controller
    context.context().context_switches.pop()

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/eager/context.py", line 136, in pop
    self.stack.pop()

IndexError: pop from empty list


         [[Node: PyFunc = PyFunc[Tin=[DT_INT64], Tout=[DT_FLOAT, DT_FLOAT], token="pyfunc_7"](arg0)]]
         [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[?], [?]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](OneShotIterator)]]
         [[Node: IteratorGetNext/_19 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_6_IteratorGetNext", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 72, in <module>
    for item in model2.predict(model2_input_fn):
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 509, in predict
    preds_evaluated = mon_sess.run(predictions)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 567, in run
    run_metadata=run_metadata)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1043, in run
    run_metadata=run_metadata)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1134, in run
    raise six.reraise(*original_exc_info)
  File "/usr/lib64/python3.5/site-packages/six.py", line 693, in reraise
    raise value
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1119, in run
    return self._sess.run(*args, **kwargs)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 1191, in run
    run_metadata=run_metadata)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 971, in run
    return self._sess.run(*args, **kwargs)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: IndexError: pop from empty list
Traceback (most recent call last):

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/ops/script_ops.py", line 157, in __call__
    ret = func(*args)

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 382, in generator_py_func
    values = next(generator_state.get_iterator(iterator_id))

  File "test.py", line 36, in gen
    for item in inputs:

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/estimator/estimator.py", line 519, in predict
    for key, value in six.iteritems(preds_evaluated)

  File "/usr/lib64/python3.5/contextlib.py", line 66, in __exit__
    next(self.gen)

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/framework/ops.py", line 5267, in get_controller
    context.context().context_switches.pop()

  File "/home/miacro/.local/lib64/python3.5/site-packages/tensorflow/python/eager/context.py", line 136, in pop
    self.stack.pop()

IndexError: pop from empty list


         [[Node: PyFunc = PyFunc[Tin=[DT_INT64], Tout=[DT_FLOAT, DT_FLOAT], token="pyfunc_7"](arg0)]]
         [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[?], [?]], output_types=[DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](OneShotIterator)]]
         [[Node: IteratorGetNext/_19 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_6_IteratorGetNext", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

@aselle aselle assigned mrry and unassigned aselle Jul 14, 2018

@aselle

This comment has been minimized.

Copy link
Member

commented Jul 14, 2018

@mrry, could you please take a look? (Are you the right person)?

@mrry mrry assigned ispirmustafa and unassigned mrry Jul 15, 2018

@mrry

This comment has been minimized.

Copy link
Member

commented Jul 15, 2018

Thanks for the straightforward reproducer, @miacro. It looks like this might be a bug in the Estimator.predict() implementation (or at least something that deserves a better error message). In particular, if I changed the following lines of code:

model1_output = model1.predict(input_fn=get_input_fn([1, 2, 3, 4]))
model2_input_fn = get_input_fn(model1_output)
for item in model2.predict(model2_input_fn):
    print(item)

...to the following:

model1_output = model1.predict(input_fn=get_input_fn([1, 2, 3, 4]))
model1_output = list(model1_output)
model2_input_fn = get_input_fn(model1_output)
for item in model2.predict(model2_input_fn):
    print(item)

...the program behaves correctly.

I'll assign this to @ispirmustafa, since he made some changes to context handling in Estimator.predict() in 0385bfe, and this may account for the error message about "context switches".

@tensorflowbutler

This comment has been minimized.

Copy link
Member

commented Aug 31, 2018

Nagging Assignee @ispirmustafa: It has been 46 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@tensorflowbutler

This comment has been minimized.

Copy link
Member

commented Sep 15, 2018

Nagging Assignee @ispirmustafa: It has been 61 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@ispirmustafa

This comment has been minimized.

Copy link
Contributor

commented Sep 17, 2018

Estimator.predict yields predictions under a graph context. The way with handle graph context in TF make yielding an issue. For example following code has same error.

I'll update the documentation of predict:

def f1():
  with tf.Graph().as_default():
    yield tf.no_op()

def f2():
  with tf.Graph().as_default():
    yield tf.no_op()

y1, y2 = f1(), f2()

next(y1)
next(y2)
next(y1)
@ispirmustafa

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.