(None, 22) Tensor is is not an element of this graph, when trying to get examples/rl_with_open_ai_gym_wrapper.py to work

[EllangoK]

Hi, I was trying to run examples/rl_with_open_ai_gym_wrapper.py on an Ubuntu 20.04 system natively just from terminal, and I keep on running into this error

ValueError: Tensor Tensor("dense_2/BiasAdd:0", shape=(None, 22), dtype=float32) is not an element of this graph.

from here

Training for 10000 steps ...
Interval 1 (0 steps performed)
/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/keras/engine/training.py:2470: UserWarning: `Model.state_updates` will be removed in a future version. This property should not be used in TensorFlow 2.0, as `updates` are applied automatically.
  warnings.warn('`Model.state_updates` will be removed in a future version. '
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/poke_env/player/env_player.py", line 363, in <lambda>
    target=lambda: env_algorithm_wrapper(self, env_algorithm_kwargs)
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/poke_env/player/env_player.py", line 347, in env_algorithm_wrapper
    env_algorithm(player, **kwargs)
  File "/home/karna/Documents/poke-net/net/test2.py", line 79, in dqn_training
    dqn.fit(player, nb_steps=nb_steps)
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/rl/core.py", line 168, in fit
    action = self.forward(observation)
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/rl/agents/dqn.py", line 224, in forward
    q_values = self.compute_q_values(state)
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/rl/agents/dqn.py", line 68, in compute_q_values
    q_values = self.compute_batch_q_values([state]).flatten()
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/rl/agents/dqn.py", line 63, in compute_batch_q_values
    q_values = self.model.predict_on_batch(batch)
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/keras/engine/training_v1.py", line 1200, in predict_on_batch
    self._make_predict_function()
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/keras/engine/training_v1.py", line 2070, in _make_predict_function
    self.predict_function = backend.function(
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/keras/backend.py", line 4092, in function
    return GraphExecutionFunction(
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/keras/backend.py", line 3885, in __init__
    with tf.control_dependencies([self.outputs[0]]):
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 5394, in control_dependencies
    return get_default_graph().control_dependencies(control_inputs)
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 4848, in control_dependencies
    c = self.as_graph_element(c)
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3759, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/home/karna/.pyenv/versions/3.9.7/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3838, in _as_graph_element_locked
    raise ValueError("Tensor %s is not an element of this graph." % obj)
ValueError: Tensor Tensor("dense_2/BiasAdd:0", shape=(None, 22), dtype=float32) is not an element of this graph.

I thought this was a compatability error, so I tried to install older tensorflow versions as I saw this was using tensorflow==2.0.0b1 and they mentioned it working.

However, I never actually managed to get tensorflow==2.0.0b1. I did get 2.0.0 though in Python 3.7.12, but I get the same issue.

  File "/home/karna/.pyenv/versions/3.7.12/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3689, in _as_graph_element_locked
    raise ValueError("Tensor %s is not an element of this graph." % obj)
ValueError: Tensor Tensor("dense_2/BiasAdd:0", shape=(None, 22), dtype=float32) is not an element of this graph.

My installed packages with python 3.9.7, let me know if what you want the 3.7.12 config, but I don't think the tensor flow version is necessarily the issue.

astunparse==1.6.3
cachetools==4.2.4
certifi==2021.10.8
charset-normalizer==2.0.7
clang==5.0
cloudpickle==2.0.0
flatbuffers==1.12
gast==0.4.0
google-auth==2.3.3
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.41.1
gym==0.21.0
h5py==3.1.0
idna==3.3
keras==2.6.0
Keras-Preprocessing==1.1.2
keras-rl2==1.0.5
Markdown==3.3.4
numpy==1.19.5
oauthlib==3.1.1
opt-einsum==3.3.0
orjson==3.6.4
poke-env==0.4.20
protobuf==3.19.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
requests==2.26.0
requests-oauthlib==1.3.0
rsa==4.7.2
six==1.15.0
tabulate==0.8.9
tensorboard==2.7.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.0
tensorflow==2.6.1
tensorflow-estimator==2.6.0
termcolor==1.1.0
typing-extensions==3.7.4.3
urllib3==1.26.7
websockets==10.0
Werkzeug==2.0.2
wrapt==1.12.1

[hsahovic]

Hi @EllangoK,

Can you try pinning keras-rl to version 1.0.3?

[DableUTeeF]

I use keras-rl2 version 1.0.3 with tensorflow 2.0.0 to 2.4.1 and get this error

  File "/home/palm/PycharmProjects/poke-env/examples/rl_with_open_ai_gym_wrapper.py", line 135, in <module>
    enable_double_dqn=True,
  File "/home/palm/miniconda3/lib/python3.6/site-packages/rl/agents/dqn.py", line 107, in __init__
    if hasattr(model.output, '__len__') and len(model.output) > 1:
  File "/home/palm/miniconda3/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 733, in __len__
    "shape information.".format(self.name))
TypeError: len is not well defined for symbolic Tensors. (dense_2/Identity:0) Please call `x.shape` rather than `len(x)` for shape information.

[DableUTeeF]

This solve the issue on my TF 2.4.0 https://stackoverflow.com/a/66955935/9477338

[hsahovic]

I'm glad to hear it. I will add revisiting this example / the docs to my todo list.

[DableUTeeF]

@hsahovic Actually I'm not the original asker. Not sure if my solution work for him or not.

[hsahovic]

Oops, my bad. Sorry about that!
@EllangoK can you try with keras-rl==1.0.3 and python 3.6.X?

[EllangoK]

How are you getting keras-rl2==1.0.3 installed, and also you meant rl2 not rl right as I can't find an keras-rl==1.0.3?

I keep on getting this error, and like I previously mentioned I never figured out how you got tensorflow==2.0.0-beta, I can only ever got 2.0.0.
Also doesn't really make sense since @DableUTeeF said he was using keras-rl2=1.0.3 but TF 2.4.0

❯ pip install keras-rl2==1.0.3
Collecting keras-rl2==1.0.3
  Using cached keras-rl2-1.0.3.tar.gz (40 kB)
ERROR: Could not find a version that satisfies the requirement tensorflow==2.0.0-beta1 (from keras-rl2)
ERROR: No matching distribution found for tensorflow==2.0.0-beta

[DableUTeeF]

@EllangoK I use pip install --no-deps keras-rl2==1.0.3

[DableUTeeF]

The --no-deps is a shortened --no-dependencies.

You just install the package and ignore whatever the package want you to install.

[EllangoK]

@DableUTeeF thanks!

Yeah, that works combined with your earlier link of modifying the len if statement in DQN agent, I really appreciate it.

[hsahovic]

Thanks for both of your feedbacks - I'll revisit this example as it's clearly outdated.