🚀 AI-HAND Isaac Copilot

This package provides an interface to control a Doosan robotic arm in IsaacSim with APIs for task management, trajectory planning, and gripper control. The copilot_ui.html provides a web interface to trigger these APIs, making simulation control easy and efficient.

---

📚 Index

• 📦 Clone the Repository
• 🛠️ Setting Up the Extension
• 📡 API Interaction Through Web Interface
• 🌐 Accessing the Web Interface
• 📡 Available API Endpoints
• 🔍 Fetching Task Data (/get_task_data)
• 📚 Parameter Details for get_task_data
• 🎮 Web Interface Usage

---

📦 Clone the Repository

To clone the repository, run:

git clone <repository_url>

---

🛠️ Setting Up the Extension

1. Open IsaacSim.
2. Navigate to Window > Extensions.
3. Click on the three lines (menu) icon and select Settings.
4. Click the plus icon and enter the path to the extension:

   ~/Isaacsim-AIHand/robotics/ai/sim/exts
   

5. Search for AI-HAND.
6. Select the THIRD PARTY tab.
7. Enable the extension by checking DISABLED and AUTOLOAD.

Installing Dependencies

To install the necessary dependencies, use the following commands:

~/.local/share/ov/pkg/isaac-sim-4.2.0/python.sh -m pip install python-xlib
~/.local/share/ov/pkg/isaac-sim-4.2.0/python.sh -m pip install pyautogui
~/.local/share/ov/pkg/isaac-sim-4.2.0/python.sh -m pip install openai

---

📡 API Interaction Through Web Interface

The APIs can be accessed through a web interface hosted by the extension at:

http://0.0.0.0/simple_ui

✅ Key Features:

• Load Scenarios: Start a scenario with a single click.
• Plan and Execute Tasks: Plan a trajectory and execute the task in the simulation.
• Move to Home and Toggle Gripper: Send the robot to the home position and toggle the gripper.
• Update Task/Scenario Name: Dynamically update the task and scenario configurations.
• Visualize Trajectories: Enable gripper and trajectory co-pilot for enhanced visual feedback.

---

🌐 Accessing the Web Interface

URL:

http://0.0.0.0/simple_ui

• The interface is automatically available when the extension starts.
• Access it in your browser to control and interact with the robot.

---

📡 Available API Endpoints

Method	Endpoint	Description
PUT	/load_scenario	Load the selected scenario.
PUT	/move_to_home	Move robot to home position.
PUT	/reset	Reset system state.
PUT	/plan	Plan trajectory based on selected task.
PUT	/execute	Execute planned trajectory.
PUT	/enable_gripper_copilot	Enable gripper co-pilot.
PUT	/enable_trajectory_copilot	Enable trajectory co-pilot.
PUT	/select_move	Select move operation.
PUT	/select_rotate	Select rotate operation.
PUT	/task_name	Set task name for execution.
PUT	/scenario_name	Set scenario name.
PUT	/cobot_config	Update cobot IP address and port.
GET	/get_task_data	Retrieve current task data from the robot.

---

🔍 Fetching Task Data (/get_task_data)

Endpoint:

GET /get_task_data

Response Format:

{
  "objects_to_spawn": {
    "ketchup": [-0.336, -0.589, 0.174],
    "slice of toast": [-0.438, -0.785, 0.147]
  },
  "fundamental_actions": {
    "grasp": {
      "is_trajectory": false,
      "coordinates": [-0.336, -0.589, 0.144],
      "csv_data": "",
      "gripper_mode_post_action": 2,
      "grasp_mode": "HORIZONTAL"
    },
    "pour": {
      "is_trajectory": true,
      "coordinates": [-0.438, -0.785, 0.117],
      "csv_data": "X,Y,Z,C\n-362.5,-614.7,120.05,175.4\n-362.9,-614.2,121.3,177.0",
      "gripper": 0,
      "grasp_mode": "HORIZONTAL"
    }
  }
}

---

📚 Parameter Details for get_task_data

1. objects_to_spawn

• Type: Dictionary
• Description: Defines objects that need to be spawned in the simulation.
• Key: Object name (e.g., ketchup, slice of toast).
• Value: List of coordinates [X, Y, Z] in meters.

---

2. fundamental_actions

• Type: Dictionary
• Description: Specifies actions that the robot can perform, including grasp and pour.
• Key: Task name (e.g., grasp, pour).
• Value: Task parameters to define the action.

---

🎯 Parameters in fundamental_actions

📌 is_trajectory

• Type: Boolean
• Description: Indicates whether the action is a trajectory-based task (true) or a position-based task (false).

📍 coordinates

• Type: List[float, float, float]
• Description: Target coordinates [X, Y, Z] in meters.
• Used if is_trajectory is false.

📄 csv_data

• Type: String
• Description: CSV-formatted string defining trajectory waypoints.
• Required if is_trajectory is true.

🤖 gripper_mode_post_action

• Type: Integer
• Description: Determines gripper action after task completion.
• Possible values:
  • 0 - No change to gripper state.
  • 1 - Open gripper after task.
  • 2 - Close gripper after task.

🖐️ grasp_mode

• Type: String
• Description: Defines how the gripper should approach the object.
• Possible values:
  • "HORIZONTAL" – Gripper approaches horizontally.
  • "VERTICAL" – Gripper approaches vertically.

---

🎮 Web Interface Usage

1. Load Scenario

• Click on Load Scenario to start the selected scenario.
• API: PUT /load_scenario

2. Reset Simulation

• Click on Reset to reset the simulation to the initial state.
• API: PUT /reset

3. Plan and Execute Tasks

• Use Plan to plan a trajectory and Execute to execute it.
• APIs:
  • PUT /plan
  • PUT /execute

4. Move to Home Position

• Move the robot to its home position.
• API: PUT /move_to_home

5. Toggle Gripper

• Toggle the gripper state (open/close).
• API: PUT /toggle_gripper

6. Update Task and Scenario

• Update task or scenario name dynamically.
• APIs:
  • PUT /task_name
  • PUT /scenario_name

7. Enable Gripper and Trajectory Co-Pilot

• Enable real-time gripper and trajectory visualization.
• APIs:
  • PUT /enable_gripper_copilot
  • PUT /enable_trajectory_copilot

---

💡 Hints and Tips

⚡ Performance Tip:

• Limit trajectory points to avoid high computation times.
• Use downsampling to reduce trajectory data.

⚠️ Caution:

• Ensure cobot IP and port are correctly set before execution.

🔍 Troubleshooting:

• Check /docs for API details.
• Inspect copilot_server.log for errors.

---