[RLlib] Getting Started first example executed with TF2, is followed by an error

[Deonixlive]

What happened + What you expected to happen

I tried the getting started commands at https://docs.ray.io/en/latest/rllib/rllib-training.html
With pip install tensorflow[and-cuda] followed by pip install "ray[rllib]".

Then I tried the example: rllib train --algo DQN --env CartPole-v1 --framework tf2 --stop '{"training_iteration": 30}'
This is followed by an ValueError instead of a saved checkpoint with a trained model.

(DQN pid=210953) Exception raised in creation task: The actor died because of an error raised in its creation task, ray::DQN.init() (pid=210953, ip=192.168.1.207, actor_id=94093f51e79110d273a302e501000000, repr=DQN)
(DQN pid=210953) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(DQN pid=210953) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py", line 554, in init
(DQN pid=210953) super().init(
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/tune/trainable/trainable.py", line 158, in init
(DQN pid=210953) self.setup(copy.deepcopy(self.config))
(DQN pid=210953) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/algorithms/algorithm.py", line 640, in setup
(DQN pid=210953) self.workers = EnvRunnerGroup(
(DQN pid=210953) ^^^^^^^^^^^^^^^
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py", line 169, in init
(DQN pid=210953) self._setup(
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py", line 260, in _setup
(DQN pid=210953) self._local_worker = self._make_worker(
(DQN pid=210953) ^^^^^^^^^^^^^^^^^^
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/env/env_runner_group.py", line 1108, in _make_worker
(DQN pid=210953) worker = cls(
(DQN pid=210953) ^^^^
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/evaluation/rollout_worker.py", line 532, in init
(DQN pid=210953) self._update_policy_map(policy_dict=self.policy_dict)
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/evaluation/rollout_worker.py", line 1737, in _update_policy_map
(DQN pid=210953) self._build_policy_map(
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/evaluation/rollout_worker.py", line 1848, in _build_policy_map
(DQN pid=210953) new_policy = create_policy_for_framework(
(DQN pid=210953) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/utils/policy.py", line 138, in create_policy_for_framework
(DQN pid=210953) return policy_class(observation_space, action_space, merged_config)
(DQN pid=210953) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/policy/eager_tf_policy.py", line 167, in init
(DQN pid=210953) super(TracedEagerPolicy, self).init(*args, **kwargs)
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/policy/eager_tf_policy.py", line 429, in init
(DQN pid=210953) self.model = make_model(self, observation_space, action_space, config)
(DQN pid=210953) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/algorithms/dqn/dqn_tf_policy.py", line 181, in build_q_model
(DQN pid=210953) q_model = ModelCatalog.get_model_v2(
(DQN pid=210953) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/models/catalog.py", line 799, in get_model_v2
(DQN pid=210953) return wrapper(
(DQN pid=210953) ^^^^^^^^
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/algorithms/dqn/distributional_q_tf_model.py", line 165, in init
(DQN pid=210953) q_out = build_action_value(name + "/action_value/", self.model_out)
(DQN pid=210953) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/ray/rllib/algorithms/dqn/distributional_q_tf_model.py", line 135, in build_action_value
(DQN pid=210953) logits = tf.expand_dims(tf.ones_like(action_scores), -1)
(DQN pid=210953) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/tensorflow/python/ops/weak_tensor_ops.py", line 88, in wrapper
(DQN pid=210953) return op(*args, **kwargs)
(DQN pid=210953) ^^^^^^^^^^^^^^^^^^^
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
(DQN pid=210953) raise e.with_traceback(filtered_tb) from None
(DQN pid=210953) File "/home/dime/miniconda3/envs/rllib/lib/python3.11/site-packages/keras/src/backend/common/keras_tensor.py", line 91, in tf_tensor
(DQN pid=210953) raise ValueError(
(DQN pid=210953) ValueError: A KerasTensor cannot be used as input to a TensorFlow function. A KerasTensor is a symbolic placeholder for a shape and dtype, used when constructing Keras Functional models or Keras Functions. You can only use it as input to a Keras layer or a Keras operation (from the namespaces keras.layers and keras.operations). You are likely doing something like:
(DQN pid=210953)
(DQN pid=210953) (DQN pid=210953) x = Input(...) (DQN pid=210953) ... (DQN pid=210953) tf_fn(x) # Invalid. (DQN pid=210953)
(DQN pid=210953)
(DQN pid=210953) What you should do instead is wrap tf_fn in a layer:
(DQN pid=210953)
(DQN pid=210953) (DQN pid=210953) class MyLayer(Layer): (DQN pid=210953) def call(self, x): (DQN pid=210953) return tf_fn(x) (DQN pid=210953) (DQN pid=210953) x = MyLayer()(x) (DQN pid=210953)

Versions / Dependencies

• Ubuntu 22.04.4 LTS
• Python 3.11.0
• Tensorflow 2.16.1
• Ray 2.24.0

Reproduction script

pip install tensorflow[and-cuda]
pip install "ray[rllib]"

rllib train --algo DQN --env CartPole-v1 --framework tf2 --stop '{"training_iteration": 30}'

Issue Severity

Medium: It is a significant difficulty but I can work around it.