XArm Follow Target

An Autonomous Camera Controller Robot
Explore the docs »

Table of Contents

1. About The Project
   • Built With
2. Using Xarm Follow Target as a Third Party Extension

About The Project

This project autonomously controls a RealSense Camera mounted to an XArm Robot. The repo is comprised of several scripts, which are designed to run synchronously across several different machines (XArm Robot, Local RealSense Camera Machine, and Remote Nautilus Cluster Machine), which utilize NVidia's Isaac Sim and Intel's RealSense Camera.

First the RealSense Camera script sends positional data to the remote machine, using OpenCV and the Camera's depth sensors. Then, the custom Isaac Sim Extension runs a Follow Target task, solving the Kinematic equations, to calculate the correct orientation of the XArm. Finally, these new orientations are sent back to the XArm itself to then update the robot is real life.

This project was headed by Professor Robert Twomey at the University of Nebraska Lincoln and is intented for educational and experimental use.

(back to top)

Built With

This project was built using the following.

• XArm
• Isaac Sim
• RealSense Camera

Using XArm Follow Target as a Third Party Extension

1. Clone the repo into the directory where you would like to store Isaac Sim Extensions
2. Open Isaac Sim and go to Windows->Extensions
3. Click the Settings Icon (Gear) and add the path to the parent directory of the repo (not XArm or XArmFollowTarget). Now the XArm Follow Target Extention should show up under the Third Party Extensions.
4. Enable the XArm Follow Target Extension and check the Autoload setting. The XArm Follow Target Extension will now appear on the top menu bar of the Isaac Sim Application.
5. Click the XArm Follow Target to use the Extension

Run it on Kubernetes / Nautilus

Port Forward local ports to the Container for the realsense camera client.

• once you have a XGL container running you will need to use the kubernetes CLI to get the specific pod name. This can be done with

 kubectl get pods -n <your namespace>

once you have your pod name we can now prot forward the local ports to the container for communication,

Run

 kubectl port-forward <running XGL pod> 12345:12345 12346:12346 -n <your namespace>

We use these ports by default

Setup for Robot Connection

Install the xArm Python SDK.

conda create -n xarm python=3
conda activate xarm
conda install -c conda-forge opencv
pip install mediapipe

 cd ~/work
 git clone https://github.com/xArm-Developer/xArm-Python-SDK
 cd xArm-Python-SDK
 python setup.py install

Install cameractrls https://github.com/soyersoyer/cameractrls

sudo snap install cameractrls
snap connect cameractrls:camera

You should be good to go:

cd ~/work/xarm-omniverse/scripts/XArm/client
python face-pose-client.py --no-socket

Running Xarm 7 Standalone from the Terminal

Open three terminal windows.

1. Run Isaac Sim

this runs isaac sim from the command line:

cd /home/user/work/xarm-omniverse/scripts/XArm
/home/user/.local/share/ov/pkg/isaac_sim-2022.2.1/python.sh xarm7_follow_target_with_standalone.py

2. Run Joint Streaming

this streams joints from isaac sim to the xarm:

cd /home/user/work/xarm-omniverse/scripts/XArm/client
conda activate xarm
python stream-joints-local.py

3. Run Face Tracking

this tracks the face and updates the follow target:

cd /home/user/work/xarm-omniverse/scripts/XArm/client
conda activate xarm
python face-pose-client-local.py

Running Xarm 5 Standalone from the Terminal

Open three terminal windows.

1. Run Isaac Sim

this runs isaac sim from the command line:

cd /home/user/work/xarm-omniverse/scripts/XArm
/home/user/.local/share/ov/pkg/isaac_sim-2022.2.1/python.sh xarm5_follow_target_with_standalone.py

2. Run Joint Streaming

this streams joints from isaac sim to the xarm:

cd /home/user/work/xarm-omniverse/scripts/XArm/client
conda activate xarm
python stream-joints-xarm5.py

3. Run Face Tracking

this tracks the face and updates the follow target:

cd /home/user/work/xarm-omniverse/scripts/XArm/client
conda activate xarm
python face-pose-client-local.py

Changing Branches

Change to bodypose:

git checkout bodypose

Change to face tracking:

git checkout estimate-3d