[Bug Report] Fetch V3 initial state problem

[wjxgeorge]

If you are submitting a bug report, please fill in the following details and use the tag [bug].

Describe the bug
The recent fix to Fetch environment for fixing the reproducibility issue changed the initial position for the environment.

A previously trained RL agent using V2 environment/dataset cannot perform well in V3 environment.

A simple fix is to add back

self.data.time = self.initial_time
self.data.qpos[:] = np.copy(self.initial_qpos)
self.data.qvel[:] = np.copy(self.initial_qvel)
if self.model.na != 0:
    self.data.act[:] = None

after self._mujoco.mj_resetData(self.model, self.data) in _reset_sim(self).

This will make the robot's start position the same as described in V2's documentation.

After adding the lines back, I can reproduce the same result using V2's agent/dataset in V3 environment.

Code example
Not applicable here.

System Info
Describe the characteristic of your environment:

• gymnasium & gymnasium robotics installed through pip
• Ubuntu 22.04
• Python 3.9

Additional context
None

Checklist

• I have checked that there is no similar issue in the repo (required)

Edit:
Fixing Typos.

[wjxgeorge]

I have another semi-related issue. I couldn't find information in the issues in this repository.

Is there a reason why the documentation on https://robotics.farama.org/envs/fetch/ is missing details about all environments except MaMuJoCo?

For example, there used to be a very detailed list of information for each of FetchReach, FetchPush, etc. But now there is only a summary page for Fetch.

I apologize if this is not a good way to ask about this. I'm not sure whether it is a bug or you intended to remove those documentations.

[Kallinteris-Andreas]

I have no idea why the documentation website is broken.
#239

Meanwhile, you can check the documentation inside the glssses or use the older 1.2.4 documentation.

[Kallinteris-Andreas]

With: gymnasium_robotics==1.3.1

>>> import gymnasium
>>> import gymnasium_robotics
>>> env = gymnasium.make("FetchReach-v3")
>>> env.reset()
({'observation': array([1.4349, 0.2641, 0.786 , 0.    , 0.    , 0.    , 0.    , 0.    ,
       0.    , 0.    ]), 'achieved_goal': array([1.4349, 0.2641, 0.786 ]), 'desired_goal': array([1.34975369, 0.7356955 , 0.52467871])}, {})

With: gymnasium_robotics==1.2.4

>>> import gymnasium
>>> env = gymnasium.make("FetchReach-v2")
>>> env.reset(seed=10)
({'observation': array([ 1.34185486e+00,  7.49100508e-01,  5.34707205e-01,  2.00232294e-04,
        6.92377335e-05, -3.25336729e-06, -2.19655130e-09,  5.16581247e-06,
        4.76882452e-06, -2.31810359e-06]), 'achieved_goal': array([1.34185486, 0.74910051, 0.5347072 ]), 'desired_goal': array([1.47865553, 0.66140505, 0.63324041])}, {})
>>> env.reset(seed=2)
({'observation': array([ 1.34185486e+00,  7.49100508e-01,  5.34707205e-01,  2.00232294e-04,
        6.92377335e-05, -3.25336729e-06, -2.19655130e-09,  5.16581247e-06,
        4.76882452e-06, -2.31810359e-06]), 'achieved_goal': array([1.34185486, 0.74910051, 0.5347072 ]), 'desired_goal': array([1.27033866, 0.68864785, 0.62897467])}, {})

So the in initialization function was crearly changed in https://github.com/Farama-Foundation/Gymnasium-Robotics/pull/208/files#diff-d73c70ad338154fb075504c05a09c54ba34d56c88e187a3c054445e14b59c848

[Kallinteris-Andreas]

The problem is that mj_resetData sets qpos=mode.qpos0 and qvel=np.zero(), and qpos0 does not match initial_qpos,
initial_qvel is not 0, initial_time should not matter as the environment dynamics are time invariant and data.act should also not matter, but we should put them back anyway

>>> import gymnasium
>>> import gymnasium_robotics
>>> env = gymnasium.make("FetchReach-v3")
>>> env.unwrapped.model.qpos0
array([0.  , 0.  , 0.  , 0.  , 0.  , 0.06, 0.  , 0.  , 0.  , 0.  , 0.  ,
       0.  , 0.  , 0.  , 0.  ])
>>> env.unwrapped.initial_qpos
array([ 4.04899882e-01,  4.80000000e-01,  2.96502492e-07,  1.25575838e-03,
        1.80409533e-10,  6.00288349e-02,  9.95180437e-03, -8.25638866e-01,
       -3.62499163e-03,  1.44389366e+00,  3.05941638e-03,  9.53219673e-01,
        5.51954430e-03,  3.90054728e-04,  1.17034674e-05])
>>> env.unwrapped.initial_qvel
array([-8.30752125e-10,  1.27908440e-12,  3.06221713e-07,  3.27694956e-03,
        1.73522642e-11,  7.22106181e-05,  7.44453590e-04,  6.46080724e-03,
       -1.39011577e-03, -2.18446790e-03,  1.10296318e-03, -1.16775031e-03,
       -9.28479541e-04,  1.66389734e-04, -7.35455335e-05])
>>> env.unwrapped.initial_time
0.4000000000000003

(note: i used mujoco==3.16 to get those numbers, initial_qpos, initial_qvel, initial_time is the same)

@wjxgeorge do you want to create a PR to fix this and update to version 4, also check if other environments are also affected
Thanks

[wjxgeorge]

@Kallinteris-Andreas Sure. The fix I proposed should be a simple solution. I'm positive Fetch environments are all affected. I'll double check others' initial states against their documented and actual initial states in 1.2.4.

[dohmjan]

Related: If someone can't find the object in FetchPush-v3. It's under the table. The proposed fix brings back the expected behavior.

[PeilangLi]

I found the same issues here, all the Fetch environments seem to have incorrect robot initial position and the cube is below the table.

[wjxgeorge]

@Kallinteris-Andreas I created a PR #256 for this.

I have a sort of related question. During the testing, I noticed mujoco>=3.0.0 has a different initial state comparing to mujoco==2.3.7. This persists even for gymnasium-robotics==1.2.4. Take FetchReach-v2 as an example.

For mujoco==2.3.7, with reset seed as 10:

{'observation': array([ 1.34185486e+00,  7.49100508e-01,  5.34707205e-01,  2.00232294e-04,
         6.92377335e-05, -3.25336729e-06, -2.19655130e-09,  5.16581247e-06,
         4.76882452e-06, -2.31810359e-06]),
 'achieved_goal': array([1.34185486, 0.74910051, 0.5347072 ]),
 'desired_goal': array([1.47865553, 0.66140505, 0.63324041]),
 'qpos': array([ 4.04899884e-01,  4.80000000e-01,  2.93323657e-07,  1.25034370e-03,
         1.80459620e-10,  6.00288349e-02,  9.69311133e-03, -8.26127718e-01,
        -3.08940529e-03,  1.44487350e+00,  2.57314464e-03,  9.51995293e-01,
         5.92928233e-03,  2.00232294e-04,  6.92377335e-05]),
 'qvel': array([-7.98637134e-10,  7.57957624e-13,  3.05732293e-07,  3.27776687e-03,
         1.62474191e-11,  7.22106030e-05,  7.08900974e-04,  6.56076483e-03,
        -1.37930223e-03, -5.09969908e-04,  1.17863420e-03, -4.04401017e-03,
        -9.87062137e-04,  1.19220613e-04, -5.79525899e-05]),
 'init_qpos': array([ 4.04899884e-01,  4.80000000e-01,  2.93323657e-07,  1.25034370e-03,
         1.80459620e-10,  6.00288349e-02,  9.69311133e-03, -8.26127718e-01,
        -3.08940529e-03,  1.44487350e+00,  2.57314464e-03,  9.51995293e-01,
         5.92928233e-03,  2.00232294e-04,  6.92377335e-05]),
 'init_qvel': array([-7.98637134e-10,  7.57957624e-13,  3.05732293e-07,  3.27776687e-03,
         1.62474191e-11,  7.22106030e-05,  7.08900974e-04,  6.56076483e-03,
        -1.37930223e-03, -5.09969908e-04,  1.17863420e-03, -4.04401017e-03,
        -9.87062137e-04,  1.19220613e-04, -5.79525899e-05])}

For mujoco==3.0.0, with reset seed as 10:

{'observation': array([ 1.34183499e+00,  7.49101034e-01,  5.34725044e-01,  3.90054728e-04,
         1.17034674e-05, -5.37468565e-06, -8.51088896e-08,  2.46630273e-05,
         6.65558934e-06, -2.94182134e-06]),
 'achieved_goal': array([1.34183499, 0.74910103, 0.53472504]),
 'desired_goal': array([1.47863577, 0.66140558, 0.63325728]),
 'qpos': array([ 4.04899882e-01,  4.80000000e-01,  2.96502492e-07,  1.25575838e-03,
         1.80409533e-10,  6.00288349e-02,  9.95180437e-03, -8.25638866e-01,
        -3.62499163e-03,  1.44389366e+00,  3.05941638e-03,  9.53219673e-01,
         5.51954430e-03,  3.90054728e-04,  1.17034674e-05]),
 'qvel': array([-8.30752125e-10,  1.27908440e-12,  3.06221713e-07,  3.27694956e-03,
         1.73522642e-11,  7.22106181e-05,  7.44453590e-04,  6.46080724e-03,
        -1.39011577e-03, -2.18446790e-03,  1.10296318e-03, -1.16775031e-03,
        -9.28479541e-04,  1.66389734e-04, -7.35455335e-05]),
 'init_qpos': array([ 4.04899882e-01,  4.80000000e-01,  2.96502492e-07,  1.25575838e-03,
         1.80409533e-10,  6.00288349e-02,  9.95180437e-03, -8.25638866e-01,
        -3.62499163e-03,  1.44389366e+00,  3.05941638e-03,  9.53219673e-01,
         5.51954430e-03,  3.90054728e-04,  1.17034674e-05]),
 'init_qvel': array([-8.30752125e-10,  1.27908440e-12,  3.06221713e-07,  3.27694956e-03,
         1.73522642e-11,  7.22106181e-05,  7.44453590e-04,  6.46080724e-03,
        -1.39011577e-03, -2.18446790e-03,  1.10296318e-03, -1.16775031e-03,
        -9.28479541e-04,  1.66389734e-04, -7.35455335e-05])}

Is this a big issue to be concerned about? The value differences appear to be on the magnitude of 1e-4 ~ 1e-5. I have some algorithms trained using V2's data but tested on corrected V3, which appear to have the same performance. Gymnasium's mujoco environments also seem to treat mujoco<3.0.0 and mujoco>=3.0.0 as the same.

[Kallinteris-Andreas]

Thanks for the PR I will check it,
also that is not a problem: https://gymnasium.farama.org/environments/mujoco/#exact-reproducibility