PPO crashes with out-of-bounds action during evaluation

[LucasBergholdt]

Description
PPO crashes during training, due to an action space violation during evaluation. The bug is a missing action clip in the evaluation loop.

The training loop clips actions (control_experiment.py:108), but the evaluation loop does not do so in either the sequential or vectorized paths (control_experiment.py:211 and 238).

PPOActorNetProbabilistic uses a plain Normal distribution with no squashing, so its outputs are unbounded.

[TwisteRmtps]

##Fix: Action Clipping in PPO Evaluation Loop

Problem

PPO crashes during training due to an action space violation in the evaluation loop.

The training loop correctly clips actions (control_experiment.py:108), but the evaluation loop does not clip actions in either the sequential or vectorized paths (control_experiment.py:211 and 238).

Since PPOActorNetProbabilistic uses a plain Normal distribution with no squashing, its outputs are unbounded causing out-of-range actions during evaluation.

---

Fix -

Add if not self._discrete_action_space: do actions = actions.clip(-1.0, 1.0) to both evaluation paths in control_experiment.py.

Vectorized path — line 208

   while not torch.all(dones):
           actions = self.agent.select_action(states, is_training=False)["action"]
           if not self._discrete_action_space:
               actions = actions.clip(-1.0, 1.0)

Sequential path — lines 236–240

while not done:
    # Select action using the agent's policy (without exploration)
        action = self.agent.select_action(state, is_training=False)["action"]
        if not self._discrete_action_space:
              action = action.clip(-1.0, 1.0)

---