Unsuccessful SAC training for demo generation

[harryzhangOG]

Hi, we made a new environment with just a floating gripper, and we are trying to generate demo data with a SAC agent. However, we found that for opening-door task, when the handle is "horizontal", SAC agent fails to succeed after training (as seen in video below, on env 1001 and 1002). I have also attached the command I was using. I switched the seed from 0 to 10 and still didn't work. Please advise if we have done anything wrong. Thanks.

python -m tools.run_rl configs/sac/sac_mani_skill_state_1M_train.py --seed=10 --cfg-options \"env_cfg.env_name={}\" \"rollout_cfg.type=Rollout\" \"rollout_cfg.num_procs=1\" \"eval_cfg.num_procs=1\" --gpu-ids=1".format(gripper_env)

[xuanlinli17]

could you repost the video? I can't see it

[harryzhangOG]

sorry, just updated with a new gif.

[xuanlinli17]

What was your reward function?

[harryzhangOG]

I just used the default dense reward. Other environments worked fine, but there are some that the agent just converged to the wrong move.

[harryzhangOG]

Is it possible to evaluate on a certain level? Just as a sanity check to see if the agent memorizes the demo data?

[xuanlinli17]

You can use env.reset(level=some_level) to reset the level to a level in demo.

(this is similar to https://github.com/haosulab/ManiSkill-Learn/blob/main/tools/convert_state.py)

[harryzhangOG]

Is there a way to do it in evaluation? We have a demo trajectory and want to train on the demo trajectory and test on the same level as a sanity check in eval. Thanks.

[xuanlinli17]

modify evaluation.py, like

ManiSkill-Learn/mani_skill_learn/env/evaluation.py

Line 115 in 8d968d2

self.recent_obs = self.env.reset()

[harryzhangOG]

ok, thanks. That's what we thought.