[question] bugs in takeoffaviary.py

[zehuiWu0306]

When I set Physics to Physics.DYN and ActionType to ActionType.DYN, the training is forced to terminate due to the nan problem
def init(self,
drone_model: DroneModel = DroneModel.CF2X,
initial_xyzs=None,
initial_rpys=None,
physics: Physics = Physics.DYN,
freq: int = 240,
aggregate_phy_steps: int = 1,
gui=False,
record=False,
obs: ObservationType = ObservationType.KIN,
act: ActionType = ActionType.DYN
):

Eval num_timesteps=2000, episode_reward=-13276.02 +/- 1389.31
Episode length: 2882.00 +/- 0.00

| eval/ | |
| mean_ep_length | 2.88e+03 |
| mean_reward | -1.33e+04 |
| time/ | |
| total_timesteps | 2000 |

New best mean reward!
D:\applications\miniconda\envs\py38\lib\site-packages\numpy\core\numeric.py:1665: RuntimeWarning: overflow encountered in multiply
multiply(a1, b2, out=cp0)
D:\applications\miniconda\envs\py38\lib\site-packages\numpy\core\numeric.py:1666: RuntimeWarning: overflow encountered in multiply
tmp = array(a2 * b1)
D:\applications\miniconda\envs\py38\lib\site-packages\numpy\core\numeric.py:1667: RuntimeWarning: invalid value encountered in subtract
cp0 -= tmp
D:\applications\miniconda\envs\py38\lib\site-packages\numpy\core\numeric.py:1668: RuntimeWarning: overflow encountered in multiply
multiply(a2, b0, out=cp1)
D:\applications\miniconda\envs\py38\lib\site-packages\numpy\core\numeric.py:1669: RuntimeWarning: overflow encountered in multiply
multiply(a0, b2, out=tmp)
D:\applications\miniconda\envs\py38\lib\site-packages\numpy\core\numeric.py:1670: RuntimeWarning: invalid value encountered in subtract
cp1 -= tmp
D:\applications\miniconda\envs\py38\lib\site-packages\numpy\core\numeric.py:1671: RuntimeWarning: overflow encountered in multiply
multiply(a0, b1, out=cp2)
D:\applications\miniconda\envs\py38\lib\site-packages\numpy\core\numeric.py:1672: RuntimeWarning: overflow encountered in multiply
multiply(a1, b0, out=tmp)
D:\applications\miniconda\envs\py38\lib\site-packages\numpy\core\numeric.py:1673: RuntimeWarning: invalid value encountered in subtract
cp2 -= tmp
Traceback (most recent call last):
File "C:\Users\zehui\Downloads\gym-pybullet-drones-master\gym-pybullet-drones-master\experiments\learning\singleagent.py", line 300, in
run(**vars(ARGS))
File "C:\Users\zehui\Downloads\gym-pybullet-drones-master\gym-pybullet-drones-master\experiments\learning\singleagent.py", line 273, in run
model.learn(total_timesteps=steps, #int(1e12),
File "D:\applications\miniconda\envs\py38\lib\site-packages\stable_baselines3\ppo\ppo.py", line 307, in learn
return super().learn(
File "D:\applications\miniconda\envs\py38\lib\site-packages\stable_baselines3\common\on_policy_algorithm.py", line 244, in learn
continue_training = self.collect_rollouts(self.env, callback, self.rollout_buffer, n_rollout_steps=self.n_steps)
File "D:\applications\miniconda\envs\py38\lib\site-packages\stable_baselines3\common\on_policy_algorithm.py", line 181, in collect_rollouts
if callback.on_step() is False:
File "D:\applications\miniconda\envs\py38\lib\site-packages\stable_baselines3\common\callbacks.py", line 100, in on_step
return self._on_step()
File "D:\applications\miniconda\envs\py38\lib\site-packages\stable_baselines3\common\callbacks.py", line 447, in _on_step
episode_rewards, episode_lengths = evaluate_policy(
File "D:\applications\miniconda\envs\py38\lib\site-packages\stable_baselines3\common\evaluation.py", line 88, in evaluate_policy
actions, states = model.predict(observations, state=states, episode_start=episode_starts, deterministic=deterministic)
File "D:\applications\miniconda\envs\py38\lib\site-packages\stable_baselines3\common\base_class.py", line 535, in predict
return self.policy.predict(observation, state, episode_start, deterministic)
File "D:\applications\miniconda\envs\py38\lib\site-packages\stable_baselines3\common\policies.py", line 343, in predict
actions = self._predict(observation, deterministic=deterministic)
File "D:\applications\miniconda\envs\py38\lib\site-packages\stable_baselines3\common\policies.py", line 687, in _predict
return self.get_distribution(observation).get_actions(deterministic=deterministic)
File "D:\applications\miniconda\envs\py38\lib\site-packages\stable_baselines3\common\policies.py", line 722, in get_distribution
return self._get_action_dist_from_latent(latent_pi)
File "D:\applications\miniconda\envs\py38\lib\site-packages\stable_baselines3\common\policies.py", line 664, in _get_action_dist_from_latent
return self.action_dist.proba_distribution(mean_actions, self.log_std)
File "D:\applications\miniconda\envs\py38\lib\site-packages\stable_baselines3\common\distributions.py", line 164, in proba_distribution
self.distribution = Normal(mean_actions, action_std)
File "D:\applications\miniconda\envs\py38\lib\site-packages\torch\distributions\normal.py", line 56, in init
super(Normal, self).init(batch_shape, validate_args=validate_args)
File "D:\applications\miniconda\envs\py38\lib\site-packages\torch\distributions\distribution.py", line 57, in init
raise ValueError(
ValueError: Expected parameter loc (Tensor of shape (1, 4)) of distribution Normal(loc: torch.Size([1, 4]), scale: torch.Size([1, 4])) to satisfy the constraint Real(), but found invalid values:
tensor([[nan, nan, nan, nan]])

i truly appreciate your help!

[JacopoPan]

Hi @zehuiWu0306

please note that there is no connection between Physics.DYN (it means the simulation update in done directly in BaseAviary and not by PyBullet) and ActionType.DYN (using an action space comprising of desired total thrust and body-frame torques). In fact, the latter is soon to be removed and replaced by the SITL build Betaflight (that takes total thrust and desired body rates as input, see branch dev-gym-v26.

From the output you posted, the numerical issues seem to be raise by torch/stable-baselines3/numpy.