Recovered copy of Oz Ramos' project for controlling UR robots with Handsfree.js.
This repository contains everything you need to start controlling your Universal Robot (UR5) handsfree through your webcam! I made this with handsfree.js during my residency at The STUDIO at CMU.
There are four main parts to this project:
Info: RoboDK supports a wide range of robots besides the UR5, so if yours is in their directory then it's a good chance that this guide will work with your robot as well!
Warning: The RoboDK simulator is very good at predicting singularities even in real time. However, it can't know about the dimensions of any attachments when calculating collisions with itself so make sure to practice without attachments first!
- Windows, OS X, Linux (I was on Windows 10)
- RoboDK >= 3.5 (I used the Trial)
- NodeJS >= 8 (I tried 8.11.4 and 10.15.1)
- Chrome or FireFox (Safari is currently broken due to a bug)
- Python >= 3.6 (I have 3.7.x)
- Make sure to have RoboDK and NodeJS installed (I used the defaults)
- Either download this project zip or run:
git clone https://github.com/labofoz/handsfree-ur5 - Inside the project directory, install dependencies with:
npm install pip install asyncio
handsfree-ur5/
├── client/ # Served with: `npm start`
│ ├── client.css
│ ├── client.js
│ ├── index.html
├── node_modules/ # Appears after you `npm install`
├── servers/
│ └── head-puppet.py # Puppeteer the robot tool as if it where a head
├── package.json
├── README.json
Because I'm not a roboticist, here's what I actually mean when I use the word:
- "Tool": refers to the final joint on the robot, where you would attach a gripper or head
- "Attachment": the actual gripper or head
- "Singularity": occurs when the robot makes a dramatic adjustment in order to continue following along a path. If you have a long attachment and a singularity occurs, there's a chance that the attachment may collide with the robot as it reorientates itself
Because we're using the RoboDK as an API to communicate with a wide range of robots, you can probably follow along with any supported robot. Just make sure to turn your robot on and manually position it to where you want the robot's origin to be.
RoboDK is a robot framework that works with several popular manufacturers out of the box, and is how we connect with our UR5. The benefit to using RoboDK is that we can live-preview what our code would look like in the simulator before actually running it on the robot.
Tip: The Trial should be sufficient since we never have to actually save anything from inside app
Once we have RoboDK installed we need to load a program. Because I don't know Python very well, what I did was create an infinitely running loop that listens for incoming WebSocket messages.
Each file in /servers/ is an individual program that can be run on the robot.
Tip: I found that keeping all the logic outside of the server led to a faster development cycle
Finally, the client is just a simple Handsfree.js plugin that sends the users current head pose to the server. Included is a quick dashboard I made to help me in developing this project.
Use npm start to run a local copy of the server.
Tip: This dashboard could also be run from your mobile device: https://handsfree-dashboard.glitch.me
This is a work in progress and you can reach me at: