Door handle slipping off of fingers

[chisarie]

Hi, I am training imitation learning policies to solve RLBench tasks. I am currently facing the following issue: even though the predicted trajectory is correct, the robot fails the task because the door handle of the fridge "slips" out of the fingers, see the gif below.

Is there any setting or change I can do to avoid or at least reduce this issue?
Thank you in advance!