On simulating Behavior tasks

[vantony1]

hi, I am trying to use OmniGibson to simulate Behavior tasks and seeing if I can use the action primitives to complete some of them. It seems like to use behavior tasks we must hand-code the list of objects that need to be rendered for each task. Is there any way to get the simulation to automatically load the objects that a given behavior task requires and then use action primitives to finish it?

The relevant example: action_primitives.wip_solve_behavior_task fails to run correctly at the moment.

/omnigibson-src/omnigibson/controllers/controller_base.py:134: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison if command_output_limits == "default" 2024-05-07 17:36:59 [14,630ms] [Warning] [omni.syntheticdata.plugin] SdRenderVarPtr missing valid input renderVar DistanceToCameraSDhost Traceback (most recent call last): File "/micromamba/envs/omnigibson/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/micromamba/envs/omnigibson/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/omnigibson-src/omnigibson/examples/action_primitives/wip_solve_behavior_task.py", line 64, in <module> main() File "/omnigibson-src/omnigibson/examples/action_primitives/wip_solve_behavior_task.py", line 42, in main env = og.Environment(configs=config) File "/omnigibson-src/omnigibson/utils/python_utils.py", line 90, in wrapper func(*values.args, **values.kwargs) File "/omnigibson-src/omnigibson/envs/env_base.py", line 77, in __init__ self.load() File "/omnigibson-src/omnigibson/envs/env_base.py", line 371, in load self._load_robots() File "/omnigibson-src/omnigibson/envs/env_base.py", line 242, in _load_robots self.scene.reset() File "/omnigibson-src/omnigibson/scenes/scene_base.py", line 493, in reset self.load_state(self._initial_state) File "/omnigibson-src/omnigibson/utils/python_utils.py", line 508, in load_state self._load_state(state=state) File "/omnigibson-src/omnigibson/scenes/scene_base.py", line 681, in _load_state self._registry.load_state(state=state, serialized=False) File "/omnigibson-src/omnigibson/utils/python_utils.py", line 508, in load_state self._load_state(state=state) File "/omnigibson-src/omnigibson/utils/registry_utils.py", line 340, in _load_state obj.load_state(state[obj.name], serialized=False) File "/omnigibson-src/omnigibson/utils/python_utils.py", line 508, in load_state self._load_state(state=state) File "/omnigibson-src/omnigibson/utils/registry_utils.py", line 340, in _load_state obj.load_state(state[obj.name], serialized=False) File "/omnigibson-src/omnigibson/utils/python_utils.py", line 508, in load_state self._load_state(state=state) File "/omnigibson-src/omnigibson/robots/manipulation_robot.py", line 1424, in _load_state for arm in state["ag_obj_constraint_params"].keys(): KeyError: 'ag_obj_constraint_params'

[vantony1]

I merged the action_primitives.wip_solve_behavior_task and environments.behavior_env_demo examples to try to get action primitives to perform actions to complete a behavior task however I am getting the following error. could anyone please give me some pointers on how to resolve this:

Executing controller Traceback (most recent call last): File "/home/test.py", line 71, in <module> main() File "/home/test.py", line 57, in main execute_controller(controller.apply_ref(StarterSemanticActionPrimitiveSet.GRASP, grasp_obj), env) File "/home/test.py", line 16, in execute_controller for action in ctrl_gen: File "/omnigibson-src/omnigibson/action_primitives/starter_semantic_action_primitives.py", line 438, in apply_ref yield from ctrl(*args) File "/omnigibson-src/omnigibson/action_primitives/starter_semantic_action_primitives.py", line 647, in _grasp yield from self._execute_release() File "/omnigibson-src/omnigibson/action_primitives/starter_semantic_action_primitives.py", line 1223, in _execute_release action = self._empty_action() File "/omnigibson-src/omnigibson/action_primitives/starter_semantic_action_primitives.py", line 1330, in _empty_action assert self.robot._controller_config["arm_" + self.arm]["mode"] == "pose_absolute_ori", "Controller must be in pose_absolute_ori mode" AssertionError: Controller must be in pose_absolute_ori mode

[Arpitrf]

You might want to confirm in your config file (something.yaml) the mode of the arm. It should be:

arm_{left/right}:
        name: InverseKinematicsController
        mode: pose_absolute_ori