Manual Control of Robot Arm By Human/User

[HarryVid]

Is it possible or is there some way to control the robots manually in the panda gym environment in order to capture recordings of demonstrations.

By manually I mean, instead of an agent predicting actions at every step, is it possible or is there some way for a human/user to control the robot using keyboard mappings or something like that.

[qgallouedec]

Hi, @Vidharth

Yes, you can manually control the robot by passing your own action into the step() method. Here is a simple example:

import gym
import panda_gym

env = gym.make('PandaPickAndPlace-v1')
env.reset()

env.step([1, 0, 0, 0]) # Go FORWARD
env.step([-1, 0, 0, 0]) # Go BACKWARD
env.step([0, 1, 0, 0]) # Go LEFT
env.step([0, -1, 0, 0]) # Go RIGHT
env.step([0, 0, 1, 0]) # Go UP
env.step([0, 0, -1, 0]) # Go DOWN
env.step([0, 0, 0, 1]) # OPEN fingers
env.step([0, 0, 0, -1]) # CLOSE fingers

Another example, where action depends on the observation:

import gym
import panda_gym

env = gym.make('PandaPush-v1')
obs = env.reset()

# Get specific observations
ee_position = obs['observation'][0:3]
object_position = obs['observation'][7:10]

# Choose action according to object position and end-effector position
action = object_position - ee_position # move in the direction of the object

# Step
obs, reward, done, info = env.step(action)

Each envrionment has its own observation and action space. Read the following for more details.

You should be able to adapt this code to do keyboard-control, if that's what you want to do (using the keyboard module for example).

For PandaReach-v1:

Observation:

• end-effector position (3 coordinates)
• end-effector velocity (3 coordinates)

Action:

• displacement (3 coordinates)

For PandaPush-v1 and PandaSlide-v1:

Observation:

• end-effector position (3 coordinates)
• end-effector velocity (3 coordinates)
• object position (3 coordinates)
• object rotation (3 coordinates)
• object linerar velocity (3 coordinates)
• object angular velocity (3 coordinates)

Action:

• displacement (3 coordinates)

For PandaPickAndPlace-v1:

Observation:

• end-effector position (3 coordinates)
• end-effector velocity (3 coordinates)
• finger width (1 coordinate)
• object position (3 coordinates)
• object rotation (3 coordinates)
• object linerar velocity (3 coordinates)
• object angular velocity (3 coordinates)

Action:

• displacement (3 coordinates)
• displacement of fingers (1 coordinate)

For PandaStack-v1:

Observation:

• end-effector position (3 coordinates)
• end-effector velocity (3 coordinates)
• finger width (1 coordinate)
• object 1 position (3 coordinates)
• object 1 rotation (3 coordinates)
• object 1 linerar velocity (3 coordinates)
• object 1 angular velocity (3 coordinates)
• object 2 position (3 coordinates)
• object 2 rotation (3 coordinates)
• object 2 linerar velocity (3 coordinates)
• object 2 angular velocity (3 coordinates)

Action:

• displacement (3 coordinates)
• displacement of fingers (1 coordinate)

EDIT: added finger width in PandaStack

[cchristofi]

Hey @qgallouedec,

If the position of the end-effector is positioned in the observation space at positions [0:3] and for the object at positions [7:10] for tasks PandaPush and PandaSlide, where are the ee-position, object position and finger width in PandaPickAndPlace and PandaStack tasks?

[qgallouedec]

You can infer it the same way from my previous comment:

	Task	Position
ee position	PandaPickAndPlace	[0:3]
	PandaStack	[0:3]
object position	PandaPickAndPlace	[0:3]
	PandaStack	[7:10] for 1, [19:22] for 2
finger width	PandaPickAndPlace	[6]
	PandaStack	[6]