Add nvidia/Isaac-GR00T into dora! #892
Description
In order to run Nvidia/isaac-gr00t we need to create a dora-gr00t node that will be able to map dora topic into groot get_action dictionary input.
The example is very clear: https://github.com/NVIDIA/Isaac-GR00T/blob/main/scripts/inference_service.py
The idea is that when we run:
- id: dora-phi4
build: pip install -e ../../node-hub/dora-phi4
path: dora-phi4
inputs:
video.ego_view: camera/image
state.left_arm: robot_left/pose
state.right_arm: robot_right/pose
state.left_hand: robot_left_hand/pose
state.right_hand: robot_right_hand/pose
state.waist: waist/pose
annotation.human.action.task_description: whisper/text
outputs:
- action.left_arm
- action.right_arm
- action.left_hand
- action.right_hand
- action.waistAnd, then call the model .get_policy() action:
obs = {
"video.ego_view": np.random.randint(
0, 256, (1, 256, 256, 3), dtype=np.uint8
),
"state.left_arm": np.random.rand(1, 7),
"state.right_arm": np.random.rand(1, 7),
"state.left_hand": np.random.rand(1, 6),
"state.right_hand": np.random.rand(1, 6),
"state.waist": np.random.rand(1, 3),
"annotation.human.action.task_description": ["do your thing!"],
}
action = policy_client.get_action(obs)