Control mode conversion does not make use of 'use-env-states' flag contrary to information provided in documentation

[ErikKrauter]

I want to change the control mode for demonstrations provided for the TurnFaucet-v0 task from the original pd_joint_pos to pd_ee_delta_pose. Thereafter I will use the script provided in ManiSkill2-Learn to do the observation mode conversion.

In the documentation under the topic of "Demonstrations" it says:

"Since some demonstrations are collected in a non-quasi-static way (objects are not fixed relative to the manipulator during manipulation) for some challenging tasks (e.g., TurnFaucet and tasks migrated from ManiSkill1), replaying actions can fail due to non-determinism in simulation. Thus, replaying trajectories by environment states is required (passing --use-env-states)."

When checking the code in replay_trajectory.py, however, one can see that the statement controlling the use of environment states for replaying trajectories is never reached. It is only ever reached if no control conversion is made, meaning if target_control_mode is None.

My question is:
When converting the control mode some of the trajectories fail, meaning they do not reach a successful goal state. It could be due to the above mentioned observation.
Is this the desired behavior?

[StoneT2000]

Thanks for bringing up the issue. In the code currently env state is not used if control modes are converted. I'll see if I can make a fix for it although I will be busy this week.

Feel free to make a PR to fix it though! I think the correct solution is to add the env state setting to all of the functions that convert control modes after calling env.step.

[StoneT2000]

Actually this has been "fixed" in the sense that control mode conversion cannot use env states (and a error is raised now if you try to do that). It is not possible to use env states to change control modes as it can take multiple steps to convert one action to another