TypeError in ray

[ErwinLiYH]

When I run this code:

from marllib import marl
# prepare env
env = marl.make_env(environment_name="mpe", map_name="simple_spread")
# initialize algorithm with appointed hyper-parameters
mappo = marl.algos.mappo(hyperparam_source="mpe")
# build agent model based on env + algorithms + user preference
model = marl.build_model(env, mappo, {"core_arch": "mlp", "encode_layer": "128-256"})
# start training
mappo.fit(env, model, stop={"timesteps_total": 1000000}, checkpoint_freq=100, share_policy="group", local_mode=False, num_gpus=1)

It will raise TypeError in the iteration 49 like:

== Status ==
Current time: 2023-12-07 17:18:19 (running for 00:04:36.77)
Memory usage on this node: 29.2/125.5 GiB
Using FIFO scheduling algorithm.
Resources requested: 2.0/24 CPUs, 1.0/1 GPUs, 0.0/60.52 GiB heap, 0.0/29.93 GiB objects (0.0/1.0 accelerator_type:G)
Result logdir: /home/yl1118/mycode/marllib_test/exp_results/mappo_mlp_simple_spread
Number of trials: 1/1 (1 RUNNING)
+--------------------------------------------+----------+---------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------+
| Trial name                                 | status   | loc                 |   iter |   total time (s) |     ts |   reward |   episode_reward_max |   episode_reward_min |   episode_len_mean |
|--------------------------------------------+----------+---------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------|
| MAPPOTrainer_mpe_simple_spread_f8b81_00000 | RUNNING  | 144.173.65.3:819841 |     49 |          266.322 | 156800 | -99.1828 |             -63.8192 |             -142.637 |                 25 |
+--------------------------------------------+----------+---------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------+


2023-12-07 17:18:20,180 ERROR trial_runner.py:924 -- Trial MAPPOTrainer_mpe_simple_spread_f8b81_00000: Error processing event.
Traceback (most recent call last):
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/tune/trial_runner.py", line 890, in _process_trial
    results = self.trial_executor.fetch_result(trial)
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/tune/ray_trial_executor.py", line 788, in fetch_result
    result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/worker.py", line 1625, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TypeError): ray::MAPPOTrainer.train() (pid=819841, ip=144.173.65.3, repr=MAPPOTrainer)
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 682, in train
    raise e
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 668, in train
    result = Trainable.train(self)
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/tune/trainable.py", line 283, in train
    result = self.step()
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/rllib/agents/trainer_template.py", line 240, in step
    evaluation_metrics = self.evaluate()
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 902, in evaluate
    self._sync_weights_to_workers(worker_set=self.evaluation_workers)
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 1007, in _sync_weights_to_workers
    worker_set.foreach_worker(lambda w: w.restore(ray.get(weights)))
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/rllib/evaluation/worker_set.py", line 181, in foreach_worker
    local_result = [func(self.local_worker())]
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/rllib/agents/trainer.py", line 1007, in <lambda>
    worker_set.foreach_worker(lambda w: w.restore(ray.get(weights)))
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/rllib/evaluation/rollout_worker.py", line 1294, in restore
    self.policy_map[pid].set_state(state)
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/rllib/policy/torch_policy.py", line 819, in set_state
    optim_state_dict = convert_to_torch_tensor(
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/rllib/utils/torch_ops.py", line 128, in convert_to_torch_tensor
    return tree.map_structure(mapping, x)
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/tree/__init__.py", line 435, in map_structure
    [func(*args) for args in zip(*map(flatten, structures))])
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/tree/__init__.py", line 435, in <listcomp>
    [func(*args) for args in zip(*map(flatten, structures))])
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/rllib/utils/torch_ops.py", line 122, in mapping
    tensor = torch.from_numpy(np.asarray(item))
TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.
== Status ==
Current time: 2023-12-07 17:18:20 (running for 00:04:37.76)
Memory usage on this node: 29.2/125.5 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/24 CPUs, 0/1 GPUs, 0.0/60.52 GiB heap, 0.0/29.93 GiB objects (0.0/1.0 accelerator_type:G)
Result logdir: /home/yl1118/mycode/marllib_test/exp_results/mappo_mlp_simple_spread
Number of trials: 1/1 (1 ERROR)
+--------------------------------------------+----------+---------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------+
| Trial name                                 | status   | loc                 |   iter |   total time (s) |     ts |   reward |   episode_reward_max |   episode_reward_min |   episode_len_mean |
|--------------------------------------------+----------+---------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------|
| MAPPOTrainer_mpe_simple_spread_f8b81_00000 | ERROR    | 144.173.65.3:819841 |     49 |          266.322 | 156800 | -99.1828 |             -63.8192 |             -142.637 |                 25 |
+--------------------------------------------+----------+---------------------+--------+------------------+--------+----------+----------------------+----------------------+--------------------+
Number of errored trials: 1
+--------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| Trial name                                 |   # failures | error file                                                                                                                                      |
|--------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------|
| MAPPOTrainer_mpe_simple_spread_f8b81_00000 |            1 | /home/yl1118/mycode/marllib_test/exp_results/mappo_mlp_simple_spread/MAPPOTrainer_mpe_simple_spread_f8b81_00000_0_2023-12-07_17-13-42/error.txt |
+--------------------------------------------+--------------+-------------------------------------------------------------------------------------------------------------------------------------------------+

Traceback (most recent call last):
  File "test_mappo.py", line 9, in <module>
    mappo.fit(env, model, stop={"timesteps_total": 1000000}, checkpoint_freq=100, share_policy="group", local_mode=False, num_gpus=1)
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/marllib/marl/__init__.py", line 316, in fit
    return run_cc(self.config_dict, env_instance, model_class, stop=stop)
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/marllib/marl/algos/run_cc.py", line 194, in run_cc
    results = POlICY_REGISTRY[exp_info["algorithm"]](model, exp_info, run_config, env_info, stop_config,
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/marllib/marl/algos/scripts/mappo.py", line 112, in run_mappo
    results = tune.run(MAPPOTrainer,
  File "/home/yl1118/miniconda3/envs/marllib/lib/python3.8/site-packages/ray/tune/tune.py", line 624, in run
    raise TuneError("Trials did not complete", incomplete_trials)
ray.tune.error.TuneError: ('Trials did not complete', [MAPPOTrainer_mpe_simple_spread_f8b81_00000])

[Aequatio-Space]

I met a similar issue and bypassed it by setting evaluation_interval=False in the fit function (some additional linking logic is required in the original library). It is a makeshift solution but may help if your task is urgent :)

[Aequatio-Space]

Please see ray-project/ray#26863, you can find the solution in commits.
Apply a patch to the ray library will solve the problem.

[Theohhhu]

@Aequatio-Space Thanks for your solution!